Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 07:52:07.802145 416 test_util.cc:276] Using random seed: -1195445012
2025-06-25T07:52:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:52:07Z Disabled control of system clock
I20250625 07:52:07.836498 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 07:52:08.169415 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:08.169816 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:08.169906 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:08.178375 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:52:08.178485 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:08.178550 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:52:08.178603 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:08.188242 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:08.190001 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:08.191869 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 07:52:08.203869 424 server_base.cc:1048] running on GCE node
W20250625 07:52:08.203799 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:08.203720 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:08.203759 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:08.205096 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:08.206624 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:08.207849 424 hybrid_clock.cc:648] HybridClock initialized: now 1750837928207779 us; error 67 us; skew 500 ppm
I20250625 07:52:08.208237 424 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:08.210886 424 webserver.cc:469] Webserver started at http://127.0.104.62:39255/ using document root <none> and password file <none>
I20250625 07:52:08.211673 424 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:08.211784 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:08.212193 424 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:08.214728 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/instance:
uuid: "00a3faeffaa4422ea73b98ee74768a4a"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.215525 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal/instance:
uuid: "00a3faeffaa4422ea73b98ee74768a4a"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.221438 424 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20250625 07:52:08.224937 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:08.226225 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250625 07:52:08.226392 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
uuid: "00a3faeffaa4422ea73b98ee74768a4a"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.226593 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:08.266216 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:08.267112 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:08.267391 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:08.288647 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38477
I20250625 07:52:08.288681 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38477 every 8 connection(s)
I20250625 07:52:08.290146 424 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
I20250625 07:52:08.294160 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:08.294521 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 424
I20250625 07:52:08.294771 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal/instance
I20250625 07:52:08.297215 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:43317
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 07:52:08.300746 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:08.317198 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
W20250625 07:52:08.318364 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
W20250625 07:52:08.319912 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
I20250625 07:52:08.369264 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } attempt: 1
W20250625 07:52:08.371166 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
I20250625 07:52:08.439455 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } attempt: 2
W20250625 07:52:08.441455 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
I20250625 07:52:08.532761 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } attempt: 3
W20250625 07:52:08.534843 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
W20250625 07:52:08.643762 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:08.644124 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:08.644239 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:08.652043 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:52:08.652166 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:08.652261 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:52:08.652328 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:08.662489 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:08.664188 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:08.666141 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:52:08.677364 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:08.677420 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:08.677420 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:08.678071 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } attempt: 4
I20250625 07:52:08.678217 492 server_base.cc:1048] running on GCE node
I20250625 07:52:08.679180 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250625 07:52:08.679845 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111)
I20250625 07:52:08.680946 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:08.682170 492 hybrid_clock.cc:648] HybridClock initialized: now 1750837928682110 us; error 52 us; skew 500 ppm
I20250625 07:52:08.682526 492 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:08.685017 492 webserver.cc:469] Webserver started at http://127.0.104.61:34053/ using document root <none> and password file <none>
I20250625 07:52:08.685799 492 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:08.685895 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:08.686223 492 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:08.688773 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/instance:
uuid: "789c06e7de2842a1b40e97e0778ab6c9"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.689620 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal/instance:
uuid: "789c06e7de2842a1b40e97e0778ab6c9"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.695616 492 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250625 07:52:08.699091 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:08.700609 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250625 07:52:08.700814 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
uuid: "789c06e7de2842a1b40e97e0778ab6c9"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:08.701013 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:08.735464 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:08.736379 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:08.736663 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:08.759595 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:43317
I20250625 07:52:08.759651 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:43317 every 8 connection(s)
I20250625 07:52:08.761189 492 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/info.pb
I20250625 07:52:08.765020 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:08.766353 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 492
I20250625 07:52:08.766539 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal/instance
I20250625 07:52:08.767944 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:33457
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 07:52:08.773772 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:08.787958 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:08.797821 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
W20250625 07:52:08.798861 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
W20250625 07:52:08.800444 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:08.849802 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 1
W20250625 07:52:08.851878 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:08.920140 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 2
W20250625 07:52:08.922230 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:08.979117 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } attempt: 5
I20250625 07:52:08.986670 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
W20250625 07:52:08.988509 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:09.013458 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 3
W20250625 07:52:09.015408 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:09.039743 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 1
W20250625 07:52:09.041553 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:09.109784 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 2
W20250625 07:52:09.111443 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
W20250625 07:52:09.112999 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:09.113315 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:09.113453 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:09.121829 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:52:09.121927 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:09.122007 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:52:09.122080 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:09.131709 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:33457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:09.133996 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:09.135848 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:52:09.146421 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:09.146500 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:09.146526 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:09.147315 563 server_base.cc:1048] running on GCE node
I20250625 07:52:09.148128 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:09.149718 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:09.150923 563 hybrid_clock.cc:648] HybridClock initialized: now 1750837929150888 us; error 35 us; skew 500 ppm
I20250625 07:52:09.151284 563 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:09.154186 563 webserver.cc:469] Webserver started at http://127.0.104.60:39021/ using document root <none> and password file <none>
I20250625 07:52:09.154994 563 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:09.155094 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:09.155457 563 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:09.158156 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data/instance:
uuid: "3eb542ac24f6405687c1067f2885bec6"
format_stamp: "Formatted at 2025-06-25 07:52:09 on dist-test-slave-ggw5"
I20250625 07:52:09.158633 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 4
I20250625 07:52:09.159001 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal/instance:
uuid: "3eb542ac24f6405687c1067f2885bec6"
format_stamp: "Formatted at 2025-06-25 07:52:09 on dist-test-slave-ggw5"
W20250625 07:52:09.160286 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:33457: Network error: Client connection negotiation failed: client connection to 127.0.104.60:33457: connect: Connection refused (error 111)
I20250625 07:52:09.164388 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 07:52:09.167768 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:09.169263 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:09.169451 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal
uuid: "3eb542ac24f6405687c1067f2885bec6"
format_stamp: "Formatted at 2025-06-25 07:52:09 on dist-test-slave-ggw5"
I20250625 07:52:09.169641 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:09.188117 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:09.189029 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:09.189270 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:09.210472 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:33457
I20250625 07:52:09.210506 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:33457 every 8 connection(s)
I20250625 07:52:09.211953 563 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/data/info.pb
I20250625 07:52:09.215888 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:09.217509 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 563
I20250625 07:52:09.217715 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-2/wal/instance
I20250625 07:52:09.217675 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 3
I20250625 07:52:09.223917 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:09.236608 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:09.243094 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap starting.
I20250625 07:52:09.246413 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
I20250625 07:52:09.248126 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:09.250181 490 log.cc:826] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:09.254035 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: No bootstrap required, opened a new log
I20250625 07:52:09.256531 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Bootstrap starting.
I20250625 07:52:09.259509 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:09.260106 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.260403 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:09.260497 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3faeffaa4422ea73b98ee74768a4a, State: Initialized, Role: FOLLOWER
I20250625 07:52:09.260808 632 log.cc:826] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:09.261209 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.262182 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.262468 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.263226 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:52:09.263679 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: No bootstrap required, opened a new log
W20250625 07:52:09.270097 444 tablet.cc:2378] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:09.270416 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.270787 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:09.270900 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb542ac24f6405687c1067f2885bec6, State: Initialized, Role: FOLLOWER
I20250625 07:52:09.271884 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.272783 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.273123 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.274312 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: configured and running, proceeding with master startup.
W20250625 07:52:09.283300 653 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 07:52:09.283443 653 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 07:52:09.275545 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:43317: OK, instance_id { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" instance_seqno: 1750837928753099 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 07:52:09.284765 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 07:52:09.297871 563 master_runner.cc:186] Error getting master registration for 127.0.104.61:43317: OK, instance_id { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" instance_seqno: 1750837928753099 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250625 07:52:09.303967 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250625 07:52:09.307539 666 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250625 07:52:09.307664 666 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250625 07:52:09.413411 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:09.413748 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.415594 654 leader_election.cc:290] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477), 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317)
I20250625 07:52:09.416402 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eb542ac24f6405687c1067f2885bec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3faeffaa4422ea73b98ee74768a4a" is_pre_election: true
I20250625 07:52:09.416941 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb542ac24f6405687c1067f2885bec6 in term 0.
I20250625 07:52:09.417588 583 leader_election.cc:304] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a, 3eb542ac24f6405687c1067f2885bec6; no voters:
I20250625 07:52:09.418022 654 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:09.418195 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:09.418303 654 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:09.422217 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.423234 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eb542ac24f6405687c1067f2885bec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a3faeffaa4422ea73b98ee74768a4a"
I20250625 07:52:09.423421 465 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:09.426756 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb542ac24f6405687c1067f2885bec6 in term 1.
I20250625 07:52:09.427165 583 leader_election.cc:304] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a, 3eb542ac24f6405687c1067f2885bec6; no voters:
I20250625 07:52:09.428102 654 leader_election.cc:290] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [CANDIDATE]: Term 1 election: Requested vote from peers 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477), 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317)
I20250625 07:52:09.428510 654 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:09.428866 654 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [term 1 LEADER]: Becoming Leader. State: Replica: 3eb542ac24f6405687c1067f2885bec6, State: Running, Role: LEADER
I20250625 07:52:09.429829 654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
W20250625 07:52:09.441284 586 tablet.cc:2378] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:09.446480 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb542ac24f6405687c1067f2885bec6. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.446858 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:09.448020 670 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:52:09.450927 670 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:52:09.459520 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } attempt: 5
I20250625 07:52:09.460556 465 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Refusing update from remote peer 3eb542ac24f6405687c1067f2885bec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:09.462745 668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:09.481194 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: Bootstrap starting.
I20250625 07:52:09.490692 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb542ac24f6405687c1067f2885bec6. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.490973 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.492261 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:09.494859 671 mvcc.cc:204] Tried to move back new op lower bound from 7171432159058477056 to 7171432158954496000. Current Snapshot: MvccSnapshot[applied={T|T < 7171432159058477056}]
I20250625 07:52:09.504243 561 log.cc:826] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:09.510550 670 catalog_manager.cc:1349] Generated new cluster ID: f1c1a60bf7df437d8f4c2e6d554d8ac6
I20250625 07:52:09.510674 670 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:52:09.515587 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: No bootstrap required, opened a new log
I20250625 07:52:09.523516 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.523818 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:09.523830 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.524024 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.524797 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.525240 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:09.525401 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789c06e7de2842a1b40e97e0778ab6c9, State: Initialized, Role: FOLLOWER
I20250625 07:52:09.526221 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:09.533375 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.535801 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.535684 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eb542ac24f6405687c1067f2885bec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:09.535633 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:52:09.536228 536 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3eb542ac24f6405687c1067f2885bec6 in term 0.
I20250625 07:52:09.535634 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3eb542ac24f6405687c1067f2885bec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9"
I20250625 07:52:09.536975 535 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:09.540669 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3eb542ac24f6405687c1067f2885bec6 in term 1.
I20250625 07:52:09.541582 534 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [term 1 FOLLOWER]: Refusing update from remote peer 3eb542ac24f6405687c1067f2885bec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:09.545640 668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:09.559031 670 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:52:09.560883 670 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:52:09.597306 670 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Generated new TSK 0
I20250625 07:52:09.598466 670 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:52:09.607764 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb542ac24f6405687c1067f2885bec6. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.608042 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.608318 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.608543 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:09.608436 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:09.608673 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9 [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:09.610479 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 07:52:09.628999 694 catalog_manager.cc:1261] Loaded cluster ID: f1c1a60bf7df437d8f4c2e6d554d8ac6
I20250625 07:52:09.629122 694 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: loading cluster ID for follower catalog manager: success
I20250625 07:52:09.634148 694 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: acquiring CA information for follower catalog manager: success
I20250625 07:52:09.642812 694 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 789c06e7de2842a1b40e97e0778ab6c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250625 07:52:10.085711 691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:10.086069 691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:10.086205 691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:10.094836 691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:10.095093 691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250625 07:52:10.098620 699 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:10.098845 699 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:10.100977 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:10.101157 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb542ac24f6405687c1067f2885bec6" is_pre_election: true
I20250625 07:52:10.101975 699 leader_election.cc:290] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317), 3eb542ac24f6405687c1067f2885bec6 (127.0.104.60:33457)
I20250625 07:52:10.102047 441 leader_election.cc:304] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a; no voters: 3eb542ac24f6405687c1067f2885bec6, 789c06e7de2842a1b40e97e0778ab6c9
I20250625 07:52:10.102434 699 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 07:52:10.106499 691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:10.108114 691 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:10.110142 691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:52:10.123080 701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:10.123097 702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:10.123840 704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:10.124471 691 server_base.cc:1048] running on GCE node
I20250625 07:52:10.125192 691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:10.126327 691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:10.127542 691 hybrid_clock.cc:648] HybridClock initialized: now 1750837930127501 us; error 28 us; skew 500 ppm
I20250625 07:52:10.127892 691 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:10.130762 691 webserver.cc:469] Webserver started at http://127.0.104.1:45299/ using document root <none> and password file <none>
I20250625 07:52:10.131747 691 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:10.131867 691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:10.132272 691 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:10.136078 691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data/instance:
uuid: "191b611b772a4fda9d1cba98776138a7"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.137215 691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/instance:
uuid: "191b611b772a4fda9d1cba98776138a7"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.144369 691 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250625 07:52:10.148967 710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.151033 691 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 07:52:10.151244 691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal
uuid: "191b611b772a4fda9d1cba98776138a7"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.151522 691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:10.183373 691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:10.184307 691 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:10.184531 691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:10.185530 691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:52:10.187068 691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:52:10.187153 691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.187229 691 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:52:10.187283 691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.231180 691 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46661
I20250625 07:52:10.231362 822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46661 every 8 connection(s)
I20250625 07:52:10.232626 691 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/data/info.pb
I20250625 07:52:10.233559 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 691
I20250625 07:52:10.233783 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/instance
I20250625 07:52:10.241080 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 07:52:10.286298 653 catalog_manager.cc:1261] Loaded cluster ID: f1c1a60bf7df437d8f4c2e6d554d8ac6
I20250625 07:52:10.286422 653 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: loading cluster ID for follower catalog manager: success
I20250625 07:52:10.290742 653 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: acquiring CA information for follower catalog manager: success
I20250625 07:52:10.293574 653 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 07:52:10.296171 825 heartbeater.cc:344] Connected to a master server at 127.0.104.61:43317
I20250625 07:52:10.298913 825 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.300015 825 heartbeater.cc:507] Master 127.0.104.61:43317 requested a full tablet report, sending...
I20250625 07:52:10.302513 526 ts_manager.cc:194] Registered new tserver with Master: 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:10.302732 823 heartbeater.cc:344] Connected to a master server at 127.0.104.60:33457
I20250625 07:52:10.302829 823 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.303260 823 heartbeater.cc:507] Master 127.0.104.60:33457 requested a full tablet report, sending...
I20250625 07:52:10.304785 597 ts_manager.cc:194] Registered new tserver with Master: 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:10.306895 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:33657
I20250625 07:52:10.308561 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:10.308754 824 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.309305 824 heartbeater.cc:507] Master 127.0.104.62:38477 requested a full tablet report, sending...
I20250625 07:52:10.312165 455 ts_manager.cc:194] Registered new tserver with Master: 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:10.614635 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:10.614995 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:10.615134 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:10.624106 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:10.624323 831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250625 07:52:10.633718 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:10.635308 831 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:10.637178 831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:52:10.649020 838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:10.649222 841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:10.649019 839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:10.650796 831 server_base.cc:1048] running on GCE node
I20250625 07:52:10.651394 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:10.652324 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:10.653546 831 hybrid_clock.cc:648] HybridClock initialized: now 1750837930653485 us; error 47 us; skew 500 ppm
I20250625 07:52:10.653872 831 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:10.656200 831 webserver.cc:469] Webserver started at http://127.0.104.2:37505/ using document root <none> and password file <none>
I20250625 07:52:10.657002 831 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:10.657109 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:10.657435 831 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:10.659950 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data/instance:
uuid: "429026e8d98d46339db143f84897a092"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.660758 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/instance:
uuid: "429026e8d98d46339db143f84897a092"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.666054 831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250625 07:52:10.669492 847 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.670734 831 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 07:52:10.670892 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal
uuid: "429026e8d98d46339db143f84897a092"
format_stamp: "Formatted at 2025-06-25 07:52:10 on dist-test-slave-ggw5"
I20250625 07:52:10.671082 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:10.686663 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:10.687697 831 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:10.687942 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:10.688889 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:52:10.690626 831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:52:10.690724 831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.690832 831 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:52:10.690904 831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:10.726965 831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:45317
I20250625 07:52:10.727023 959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:45317 every 8 connection(s)
I20250625 07:52:10.728386 831 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/data/info.pb
I20250625 07:52:10.738317 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 831
I20250625 07:52:10.738587 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/instance
I20250625 07:52:10.743086 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250625 07:52:10.748165 962 heartbeater.cc:344] Connected to a master server at 127.0.104.61:43317
I20250625 07:52:10.748318 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:10.748797 962 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.748795 961 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.750123 961 heartbeater.cc:507] Master 127.0.104.62:38477 requested a full tablet report, sending...
I20250625 07:52:10.750252 960 heartbeater.cc:344] Connected to a master server at 127.0.104.60:33457
I20250625 07:52:10.750365 960 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:10.750913 960 heartbeater.cc:507] Master 127.0.104.60:33457 requested a full tablet report, sending...
I20250625 07:52:10.750147 962 heartbeater.cc:507] Master 127.0.104.61:43317 requested a full tablet report, sending...
I20250625 07:52:10.751984 526 ts_manager.cc:194] Registered new tserver with Master: 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:10.752187 455 ts_manager.cc:194] Registered new tserver with Master: 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:10.752822 597 ts_manager.cc:194] Registered new tserver with Master: 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:10.753924 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:34495
I20250625 07:52:10.810770 823 heartbeater.cc:499] Master 127.0.104.60:33457 was elected leader, sending a full tablet report...
W20250625 07:52:11.094393 967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:11.094745 967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:11.094878 967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:52:11.102761 967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:11.102986 967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250625 07:52:11.112923 967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:11.114550 967 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:11.116393 967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 07:52:11.120810 971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:11.120983 971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:11.121604 971 leader_election.cc:290] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317), 3eb542ac24f6405687c1067f2885bec6 (127.0.104.60:33457)
I20250625 07:52:11.122145 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3eb542ac24f6405687c1067f2885bec6" is_pre_election: true
I20250625 07:52:11.122186 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:11.122826 442 leader_election.cc:304] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a; no voters: 3eb542ac24f6405687c1067f2885bec6, 789c06e7de2842a1b40e97e0778ab6c9
I20250625 07:52:11.123072 971 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 07:52:11.128595 973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:11.128638 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:11.128680 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:11.129863 967 server_base.cc:1048] running on GCE node
I20250625 07:52:11.130453 967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:11.131341 967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:11.132534 967 hybrid_clock.cc:648] HybridClock initialized: now 1750837931132492 us; error 40 us; skew 500 ppm
I20250625 07:52:11.132876 967 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:11.139231 967 webserver.cc:469] Webserver started at http://127.0.104.3:34801/ using document root <none> and password file <none>
I20250625 07:52:11.140105 967 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:11.140200 967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:11.140565 967 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:52:11.143013 967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data/instance:
uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
format_stamp: "Formatted at 2025-06-25 07:52:11 on dist-test-slave-ggw5"
I20250625 07:52:11.143748 967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/instance:
uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
format_stamp: "Formatted at 2025-06-25 07:52:11 on dist-test-slave-ggw5"
I20250625 07:52:11.149044 967 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250625 07:52:11.152281 982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:11.153678 967 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:11.153841 967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal
uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
format_stamp: "Formatted at 2025-06-25 07:52:11 on dist-test-slave-ggw5"
I20250625 07:52:11.154040 967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:11.167053 967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:11.167909 967 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:11.168146 967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:11.169162 967 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:52:11.170770 967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:52:11.170866 967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:11.170976 967 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:52:11.171047 967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:11.206972 967 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36221
I20250625 07:52:11.207036 1094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36221 every 8 connection(s)
I20250625 07:52:11.208335 967 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/data/info.pb
I20250625 07:52:11.214169 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 967
I20250625 07:52:11.214354 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/instance
I20250625 07:52:11.229875 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:11.230441 1096 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:11.231642 1096 heartbeater.cc:507] Master 127.0.104.62:38477 requested a full tablet report, sending...
I20250625 07:52:11.233461 455 ts_manager.cc:194] Registered new tserver with Master: 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:11.233978 1097 heartbeater.cc:344] Connected to a master server at 127.0.104.61:43317
I20250625 07:52:11.234103 1097 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:11.234580 1097 heartbeater.cc:507] Master 127.0.104.61:43317 requested a full tablet report, sending...
I20250625 07:52:11.235472 526 ts_manager.cc:194] Registered new tserver with Master: 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:11.239698 1095 heartbeater.cc:344] Connected to a master server at 127.0.104.60:33457
I20250625 07:52:11.239851 1095 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:11.240295 1095 heartbeater.cc:507] Master 127.0.104.60:33457 requested a full tablet report, sending...
I20250625 07:52:11.241171 596 ts_manager.cc:194] Registered new tserver with Master: 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:11.242071 596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:58575
I20250625 07:52:11.254909 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:52:11.255967 960 heartbeater.cc:499] Master 127.0.104.60:33457 was elected leader, sending a full tablet report...
I20250625 07:52:11.284153 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.284245 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_09d0d6f26dac4df085712fb32e0da7c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.285682 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:11.285712 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09d0d6f26dac4df085712fb32e0da7c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.294806 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_51bb16ee197440e09ff6c6bb2b3d9fd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.295876 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51bb16ee197440e09ff6c6bb2b3d9fd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.294579 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9e12aa1b7b0a4b958e34ddb366b13b31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.297302 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e12aa1b7b0a4b958e34ddb366b13b31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.301020 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_69b92e40762c4b529d1206b05fd3e04b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.301651 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b92e40762c4b529d1206b05fd3e04b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.326249 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.326572 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_52701c8632de4fc192a1370dea25e1a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.327034 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:11.327103 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52701c8632de4fc192a1370dea25e1a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.352001 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_530a0780436e46af8dc449d3fd1ef7df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.352676 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_530a0780436e46af8dc449d3fd1ef7df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.360487 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2b0c22424fae4d6c814d98bb9d9c8c5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.361301 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b0c22424fae4d6c814d98bb9d9c8c5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.367158 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fc8e2d79650047318f5d92a228ea57bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.367911 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc8e2d79650047318f5d92a228ea57bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.371814 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_69b92e40762c4b529d1206b05fd3e04b" } new_table_name: "default.Table_ca2264fb0e544c69b428bb55a1f7c313" modify_external_catalogs: true
I20250625 07:52:11.380079 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_530a0780436e46af8dc449d3fd1ef7df" } new_table_name: "default.Table_81b293e5e11b4574876e61fa0bd02dd0" modify_external_catalogs: true
I20250625 07:52:11.385337 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_52701c8632de4fc192a1370dea25e1a1" } new_table_name: "default.Table_5e62955ce44346d290ac1a75c5daa6b6" modify_external_catalogs: true
I20250625 07:52:11.389715 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_51bb16ee197440e09ff6c6bb2b3d9fd4" } new_table_name: "default.Table_994604bdceea483cb2c00ad1e1c642c5" modify_external_catalogs: true
I20250625 07:52:11.394484 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9e12aa1b7b0a4b958e34ddb366b13b31" } new_table_name: "default.Table_d060de5813af448fa1de9410a0d6e4cd" modify_external_catalogs: true
I20250625 07:52:11.400421 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9" } modify_external_catalogs: true
I20250625 07:52:11.400664 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9" } modify_external_catalogs: true
I20250625 07:52:11.401194 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09d0d6f26dac4df085712fb32e0da7c5" } modify_external_catalogs: true
I20250625 07:52:11.401417 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09d0d6f26dac4df085712fb32e0da7c5" } modify_external_catalogs: true
I20250625 07:52:11.405882 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5" } modify_external_catalogs: true
I20250625 07:52:11.406138 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5" } modify_external_catalogs: true
W20250625 07:52:11.419205 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e162d6b0fc9e42f8b65446dc6efeb0b5 has no consensus state
W20250625 07:52:11.419291 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3471c83e015b44adbe8224c17a1c2eaf has no consensus state
W20250625 07:52:11.448995 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e5f06c153744fb7a5104d5e5ac47575 has no consensus state
W20250625 07:52:11.452149 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e90434606299495081d752df4761f9f8 has no consensus state
W20250625 07:52:11.453030 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (table default.Table_ca2264fb0e544c69b428bb55a1f7c313 [id=65a6d16ff5e444e286e0ca97322594c8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e162d6b0fc9e42f8b65446dc6efeb0b5 has no consensus state
W20250625 07:52:11.454993 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a95c624abd8a45079c89fe1750faafcf has no consensus state
I20250625 07:52:11.457399 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fc8e2d79650047318f5d92a228ea57bb" } modify_external_catalogs: true
I20250625 07:52:11.457648 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fc8e2d79650047318f5d92a228ea57bb" } modify_external_catalogs: true
I20250625 07:52:11.457798 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 688190c7d95541739902052507a884d2 (DEFAULT_TABLE table=default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.459638 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688190c7d95541739902052507a884d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.461800 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (DEFAULT_TABLE table=default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.462366 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753404a0a77c46e6a90fbacb6cdfdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.464738 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b0c22424fae4d6c814d98bb9d9c8c5a" } modify_external_catalogs: true
I20250625 07:52:11.464946 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b0c22424fae4d6c814d98bb9d9c8c5a" } modify_external_catalogs: true
I20250625 07:52:11.467757 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.468261 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e5f06c153744fb7a5104d5e5ac47575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.470186 591 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 838163185c294e588b1c3986d7902dd0
I20250625 07:52:11.473832 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (DEFAULT_TABLE table=default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.474301 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e252f6996cd42ccaa4054f0e16fcfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.475395 1023 tablet_service.cc:1468] Processing CreateTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (DEFAULT_TABLE table=default.Table_ca2264fb0e544c69b428bb55a1f7c313 [id=65a6d16ff5e444e286e0ca97322594c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.475186 895 tablet_service.cc:1468] Processing CreateTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (DEFAULT_TABLE table=default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.475982 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e162d6b0fc9e42f8b65446dc6efeb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.477097 891 tablet_service.cc:1468] Processing CreateTablet for tablet a95c624abd8a45079c89fe1750faafcf (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.477851 889 tablet_service.cc:1468] Processing CreateTablet for tablet e90434606299495081d752df4761f9f8 (DEFAULT_TABLE table=default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.478641 894 tablet_service.cc:1468] Processing CreateTablet for tablet 688190c7d95541739902052507a884d2 (DEFAULT_TABLE table=default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.479290 888 tablet_service.cc:1468] Processing CreateTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (DEFAULT_TABLE table=default.Table_ca2264fb0e544c69b428bb55a1f7c313 [id=65a6d16ff5e444e286e0ca97322594c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.480062 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf (DEFAULT_TABLE table=default.Table_81b293e5e11b4574876e61fa0bd02dd0 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.480108 893 tablet_service.cc:1468] Processing CreateTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.480576 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e90434606299495081d752df4761f9f8 (DEFAULT_TABLE table=default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.480890 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (DEFAULT_TABLE table=default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.481107 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90434606299495081d752df4761f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.483266 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753404a0a77c46e6a90fbacb6cdfdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.484728 1026 tablet_service.cc:1468] Processing CreateTablet for tablet a95c624abd8a45079c89fe1750faafcf (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.485200 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95c624abd8a45079c89fe1750faafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.485347 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e162d6b0fc9e42f8b65446dc6efeb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.485539 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90434606299495081d752df4761f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.485728 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688190c7d95541739902052507a884d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.485873 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3471c83e015b44adbe8224c17a1c2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.485811 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf (DEFAULT_TABLE table=default.Table_81b293e5e11b4574876e61fa0bd02dd0 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.486023 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95c624abd8a45079c89fe1750faafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.486166 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e252f6996cd42ccaa4054f0e16fcfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.486335 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e5f06c153744fb7a5104d5e5ac47575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.486776 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3471c83e015b44adbe8224c17a1c2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.525967 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 688190c7d95541739902052507a884d2
I20250625 07:52:11.527215 596 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 56f15db7e65f4036acc97323b402c2b5
I20250625 07:52:11.528223 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1e252f6996cd42ccaa4054f0e16fcfa5
I20250625 07:52:11.529723 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 753404a0a77c46e6a90fbacb6cdfdbe1
I20250625 07:52:11.530682 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.531569 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.531908 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.532958 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.533578 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.534063 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.546909 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.547662 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.547758 758 tablet_service.cc:1468] Processing CreateTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (DEFAULT_TABLE table=default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.549381 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_90101094ccf24d36a5fbb85a32358074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.549871 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753404a0a77c46e6a90fbacb6cdfdbe1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:11.549892 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90101094ccf24d36a5fbb85a32358074 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.554913 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_986560b167ca4d83a36975224072c523"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.555433 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_986560b167ca4d83a36975224072c523 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.555984 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_45646a782da0472da9dc6fca099117e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.556444 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45646a782da0472da9dc6fca099117e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.557288 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 4e5f06c153744fb7a5104d5e5ac47575 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:11.559310 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.559865 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.566398 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_14f45b55fa04410699168699169a93b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.567620 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6451a67575d74a62a963f0bb3358bcbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.568087 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6451a67575d74a62a963f0bb3358bcbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.569483 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4e5f06c153744fb7a5104d5e5ac47575
I20250625 07:52:11.569943 591 catalog_manager.cc:6652] ReplaceTablet: tablet 4e5f06c153744fb7a5104d5e5ac47575 deleted and replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:11.571393 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.572060 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:11.572428 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14f45b55fa04410699168699169a93b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.575457 1162 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:11.578471 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_90e4ef7124d54235bdea0b97ddaf7ecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.579134 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90e4ef7124d54235bdea0b97ddaf7ecd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.580902 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 3471c83e015b44adbe8224c17a1c2eaf from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:11.581365 757 tablet_service.cc:1468] Processing CreateTablet for tablet 688190c7d95541739902052507a884d2 (DEFAULT_TABLE table=default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.581924 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688190c7d95541739902052507a884d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.590967 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (DEFAULT_TABLE table=default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa [id=e4b3ee29eafd4621811d8db11bdd65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.591265 1163 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:11.591519 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.593043 892 tablet_service.cc:1468] Processing CreateTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (DEFAULT_TABLE table=default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa [id=e4b3ee29eafd4621811d8db11bdd65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.593497 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.595086 1163 tablet_bootstrap.cc:654] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.596387 1163 log.cc:826] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:11.600992 1162 tablet_bootstrap.cc:654] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.602541 1162 log.cc:826] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:11.603845 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (DEFAULT_TABLE table=default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.604171 755 tablet_service.cc:1468] Processing CreateTablet for tablet a95c624abd8a45079c89fe1750faafcf (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.604401 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e252f6996cd42ccaa4054f0e16fcfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.604789 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95c624abd8a45079c89fe1750faafcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.609654 1030 tablet_service.cc:1468] Processing CreateTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.610154 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 (DEFAULT_TABLE table=default.Table_986560b167ca4d83a36975224072c523 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.610415 1027 tablet_service.cc:1468] Processing CreateTablet for tablet bf0189f6651347a18663a94aa61ee267 (DEFAULT_TABLE table=default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.610136 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f3fbffc954fdb8eb5105b5d383d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.610875 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ef72e8a4f54fd4994dcf4725337478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.611059 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf0189f6651347a18663a94aa61ee267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.611097 886 tablet_service.cc:1468] Processing CreateTablet for tablet bf0189f6651347a18663a94aa61ee267 (DEFAULT_TABLE table=default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.611775 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf0189f6651347a18663a94aa61ee267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.616796 1025 tablet_service.cc:1468] Processing CreateTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 (DEFAULT_TABLE table=default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.617322 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f774b4ac0c774ee7bf33d13a678cd368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.621193 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.621799 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.624128 885 tablet_service.cc:1468] Processing CreateTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c (DEFAULT_TABLE table=default.Table_6451a67575d74a62a963f0bb3358bcbb [id=7fbebf8268534597bd15d2bc70204cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.624298 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_81b293e5e11b4574876e61fa0bd02dd0" } new_table_name: "default.Table_60d9b99aab8e4f81b656cbb342e1bed1" modify_external_catalogs: true
I20250625 07:52:11.624707 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9916612cb0234ec3a6df8f22c2b50e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.625921 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6451a67575d74a62a963f0bb3358bcbb" } new_table_name: "default.Table_9fdc22d5be4b43a4be699388015b9864" modify_external_catalogs: true
I20250625 07:52:11.628850 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4347d3ef74664e59b97f0e68e9abcfc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.629393 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4347d3ef74664e59b97f0e68e9abcfc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.629750 884 tablet_service.cc:1468] Processing CreateTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 (DEFAULT_TABLE table=default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.630239 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e14b60ee7a843e5bb878148d8fd2e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.639515 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c (DEFAULT_TABLE table=default.Table_6451a67575d74a62a963f0bb3358bcbb [id=7fbebf8268534597bd15d2bc70204cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.639989 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9916612cb0234ec3a6df8f22c2b50e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.648428 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (DEFAULT_TABLE table=default.Table_14f45b55fa04410699168699169a93b4 [id=15c681d0005742e781fe94d2c8adc9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.651346 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_14f45b55fa04410699168699169a93b4" } new_table_name: "default.Table_be980d2eeeda473dbcfc4d7f282e21a9" modify_external_catalogs: true
I20250625 07:52:11.652377 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 (DEFAULT_TABLE table=default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.651417 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3471c83e015b44adbe8224c17a1c2eaf
I20250625 07:52:11.652995 588 catalog_manager.cc:6652] ReplaceTablet: tablet 3471c83e015b44adbe8224c17a1c2eaf deleted and replaced by tablet 834d516547b046288e58b7cde4c56e5a
I20250625 07:52:11.653847 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.655519 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa" } new_table_name: "default.Table_b7677386183a479f9714604d7c2ab058" modify_external_catalogs: true
I20250625 07:52:11.656729 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca2264fb0e544c69b428bb55a1f7c313" } new_table_name: "default.Table_cea9385219bd4aadaa7ea96f51c8bfee" modify_external_catalogs: true
I20250625 07:52:11.656925 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e14b60ee7a843e5bb878148d8fd2e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.663900 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.666087 894 tablet_service.cc:1468] Processing CreateTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 (DEFAULT_TABLE table=default.Table_986560b167ca4d83a36975224072c523 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.666664 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ef72e8a4f54fd4994dcf4725337478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.672501 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_45646a782da0472da9dc6fca099117e6" } modify_external_catalogs: true
I20250625 07:52:11.672770 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_45646a782da0472da9dc6fca099117e6" } modify_external_catalogs: true
W20250625 07:52:11.674921 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9916612cb0234ec3a6df8f22c2b50e7c consensus state has no leader
W20250625 07:52:11.676029 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 834d516547b046288e58b7cde4c56e5a has no consensus state
I20250625 07:52:11.677680 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf (DEFAULT_TABLE table=default.Table_81b293e5e11b4574876e61fa0bd02dd0 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.678381 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3471c83e015b44adbe8224c17a1c2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.682587 753 tablet_service.cc:1468] Processing CreateTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (DEFAULT_TABLE table=default.Table_ca2264fb0e544c69b428bb55a1f7c313 [id=65a6d16ff5e444e286e0ca97322594c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:11.683097 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 consensus state has no leader
I20250625 07:52:11.683115 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e162d6b0fc9e42f8b65446dc6efeb0b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:11.683610 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e162d6b0fc9e42f8b65446dc6efeb0b5 consensus state has no leader
I20250625 07:52:11.685246 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.685431 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 9861eea50d944b2c92be67c3c7c34582 (DEFAULT_TABLE table=default.Table_90e4ef7124d54235bdea0b97ddaf7ecd [id=cc9a674f99634eae89fb70aad7f6ef99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.685870 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861eea50d944b2c92be67c3c7c34582. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:11.687355 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 consensus state has no leader
I20250625 07:52:11.688347 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.688737 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.689167 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.698691 750 tablet_service.cc:1468] Processing CreateTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (DEFAULT_TABLE table=default.Table_a854b1d9b7bd4bec9a4dab6adb6429aa [id=e4b3ee29eafd4621811d8db11bdd65c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.699965 749 tablet_service.cc:1468] Processing CreateTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 (DEFAULT_TABLE table=default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.699226 751 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (DEFAULT_TABLE table=default.Table_14f45b55fa04410699168699169a93b4 [id=15c681d0005742e781fe94d2c8adc9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.701232 748 tablet_service.cc:1468] Processing CreateTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.701761 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f774b4ac0c774ee7bf33d13a678cd368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.713284 890 tablet_service.cc:1468] Processing CreateTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (DEFAULT_TABLE table=default.Table_14f45b55fa04410699168699169a93b4 [id=15c681d0005742e781fe94d2c8adc9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.713832 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.714628 885 tablet_service.cc:1468] Processing CreateTablet for tablet 9861eea50d944b2c92be67c3c7c34582 (DEFAULT_TABLE table=default.Table_90e4ef7124d54235bdea0b97ddaf7ecd [id=cc9a674f99634eae89fb70aad7f6ef99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.715081 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861eea50d944b2c92be67c3c7c34582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.719461 893 tablet_service.cc:1468] Processing CreateTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 (DEFAULT_TABLE table=default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.719923 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f774b4ac0c774ee7bf33d13a678cd368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.724756 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.725190 891 tablet_service.cc:1468] Processing CreateTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.725665 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f3fbffc954fdb8eb5105b5d383d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.727006 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1f3fbffc954fdb8eb5105b5d383d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.729715 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4347d3ef74664e59b97f0e68e9abcfc0" } modify_external_catalogs: true
I20250625 07:52:11.729902 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4347d3ef74664e59b97f0e68e9abcfc0" } modify_external_catalogs: true
I20250625 07:52:11.730657 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5" } modify_external_catalogs: true
I20250625 07:52:11.730798 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5" } modify_external_catalogs: true
I20250625 07:52:11.731246 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cea9385219bd4aadaa7ea96f51c8bfee" } modify_external_catalogs: true
I20250625 07:52:11.731352 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cea9385219bd4aadaa7ea96f51c8bfee" } modify_external_catalogs: true
I20250625 07:52:11.733685 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b7677386183a479f9714604d7c2ab058" } modify_external_catalogs: true
I20250625 07:52:11.733830 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b7677386183a479f9714604d7c2ab058" } modify_external_catalogs: true
I20250625 07:52:11.744385 1095 heartbeater.cc:499] Master 127.0.104.60:33457 was elected leader, sending a full tablet report...
W20250625 07:52:11.750118 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e252f6996cd42ccaa4054f0e16fcfa5 already in progress: creating tablet
W20250625 07:52:11.750774 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 688190c7d95541739902052507a884d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688190c7d95541739902052507a884d2 already in progress: creating tablet
I20250625 07:52:11.759881 1163 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:11.764073 1163 ts_tablet_manager.cc:1397] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.173s user 0.014s sys 0.024s
I20250625 07:52:11.765097 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1af8e3ae3fb04f159b97af62f5256fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.765239 884 tablet_service.cc:1468] Processing CreateTablet for tablet 51279fb42ae8480b8511432cb4f808ab (DEFAULT_TABLE table=default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:11.765825 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1af8e3ae3fb04f159b97af62f5256fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.766162 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51279fb42ae8480b8511432cb4f808ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.770234 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 834d516547b046288e58b7cde4c56e5a (DEFAULT_TABLE table=default.Table_60d9b99aab8e4f81b656cbb342e1bed1 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.770675 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834d516547b046288e58b7cde4c56e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.773255 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e87722f6f3e42048774f5c2a33ff506"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.773797 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e87722f6f3e42048774f5c2a33ff506 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.774688 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a940f52db1ba4b5caa2137bf0827a978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.774716 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.775171 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a940f52db1ba4b5caa2137bf0827a978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:11.775229 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.780865 594 master_service.cc:948] ReplaceTablet: received request to replace tablet a95c624abd8a45079c89fe1750faafcf from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:11.791096 757 tablet_service.cc:1468] Processing CreateTablet for tablet e90434606299495081d752df4761f9f8 (DEFAULT_TABLE table=default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.791968 747 tablet_service.cc:1468] Processing CreateTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 (DEFAULT_TABLE table=default.Table_986560b167ca4d83a36975224072c523 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.792490 746 tablet_service.cc:1468] Processing CreateTablet for tablet bf0189f6651347a18663a94aa61ee267 (DEFAULT_TABLE table=default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.793071 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.795480 1163 raft_consensus.cc:357] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.795899 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 37f7c060fb8648b9aaa766bbaa1b4ba7
I20250625 07:52:11.796778 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 (DEFAULT_TABLE table=default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.796926 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_22595703a6814afeaeb032090df4c1a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.797497 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e14b60ee7a843e5bb878148d8fd2e53. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:11.797593 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22595703a6814afeaeb032090df4c1a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.798967 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:11.799767 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_70696cfa27174f038c5b358f0684dd61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.800156 1163 raft_consensus.cc:383] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:11.800336 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70696cfa27174f038c5b358f0684dd61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.800719 1163 raft_consensus.cc:738] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.800827 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.801126 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9861eea50d944b2c92be67c3c7c34582 (DEFAULT_TABLE table=default.Table_90e4ef7124d54235bdea0b97ddaf7ecd [id=cc9a674f99634eae89fb70aad7f6ef99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.801609 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:11.801740 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9861eea50d944b2c92be67c3c7c34582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.802325 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0c90f8a308d4444e8b015f34dd0c4140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.802861 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c90f8a308d4444e8b015f34dd0c4140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.803376 1163 consensus_queue.cc:260] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.804032 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bf0189f6651347a18663a94aa61ee267
I20250625 07:52:11.804818 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e162d6b0fc9e42f8b65446dc6efeb0b5
I20250625 07:52:11.805253 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94ef72e8a4f54fd4994dcf4725337478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.808971 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf0189f6651347a18663a94aa61ee267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.813941 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_601b836403094b4cb7106f74b03172e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.814484 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:11.814661 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_601b836403094b4cb7106f74b03172e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.815119 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.815722 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.816213 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.816736 886 tablet_service.cc:1468] Processing CreateTablet for tablet 834d516547b046288e58b7cde4c56e5a (DEFAULT_TABLE table=default.Table_60d9b99aab8e4f81b656cbb342e1bed1 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.817234 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834d516547b046288e58b7cde4c56e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.824223 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.824894 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 51279fb42ae8480b8511432cb4f808ab (DEFAULT_TABLE table=default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.825443 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51279fb42ae8480b8511432cb4f808ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.834067 1163 ts_tablet_manager.cc:1428] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.069s user 0.007s sys 0.011s
I20250625 07:52:11.836933 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.834600 887 tablet_service.cc:1468] Processing CreateTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f (DEFAULT_TABLE table=default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.837841 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.837996 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95fce53f9a654fcaa8a52e5733c1b74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.838732 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.844870 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.849133 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f (DEFAULT_TABLE table=default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.849753 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95fce53f9a654fcaa8a52e5733c1b74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.851032 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c (DEFAULT_TABLE table=default.Table_6451a67575d74a62a963f0bb3358bcbb [id=7fbebf8268534597bd15d2bc70204cb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.851549 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9916612cb0234ec3a6df8f22c2b50e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.846413 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.849340 1163 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:11.854611 1163 tablet_bootstrap.cc:654] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.857779 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 (DEFAULT_TABLE table=default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.858428 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e5f06c153744fb7a5104d5e5ac47575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.864666 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90434606299495081d752df4761f9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.865527 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:11.856806 891 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.878490 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1041c1357c9e43e296c85b90037c3964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:11.879092 752 tablet_service.cc:1468] Processing CreateTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f (DEFAULT_TABLE table=default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:11.879180 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1041c1357c9e43e296c85b90037c3964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.879804 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95fce53f9a654fcaa8a52e5733c1b74f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.887079 1162 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:11.887465 1162 ts_tablet_manager.cc:1397] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.312s user 0.012s sys 0.050s
I20250625 07:52:11.891450 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1a5bdf9178904b3a9d10e1621fb031e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:11.892030 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a5bdf9178904b3a9d10e1621fb031e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:11.892962 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.893469 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_986560b167ca4d83a36975224072c523" } new_table_name: "default.Table_2cba52698a89454d9ea17c33d2d93fc3" modify_external_catalogs: true
I20250625 07:52:11.893673 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.894774 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_60d9b99aab8e4f81b656cbb342e1bed1" } new_table_name: "default.Table_1d369f97c52447bf90aa1a6f964a5199" modify_external_catalogs: true
W20250625 07:52:11.896067 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 688190c7d95541739902052507a884d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688190c7d95541739902052507a884d2 already in progress: creating tablet
W20250625 07:52:11.896327 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e252f6996cd42ccaa4054f0e16fcfa5 already in progress: creating tablet
W20250625 07:52:11.896579 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753404a0a77c46e6a90fbacb6cdfdbe1 already in progress: creating tablet
I20250625 07:52:11.896842 746 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.897109 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.897594 743 tablet_service.cc:1468] Processing CreateTablet for tablet 834d516547b046288e58b7cde4c56e5a (DEFAULT_TABLE table=default.Table_60d9b99aab8e4f81b656cbb342e1bed1 [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.897943 1163 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:11.898133 1163 ts_tablet_manager.cc:1397] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.001s
I20250625 07:52:11.898334 745 tablet_service.cc:1468] Processing CreateTablet for tablet 51279fb42ae8480b8511432cb4f808ab (DEFAULT_TABLE table=default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.898597 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a95c624abd8a45079c89fe1750faafcf
I20250625 07:52:11.898734 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.899009 594 catalog_manager.cc:6652] ReplaceTablet: tablet a95c624abd8a45079c89fe1750faafcf deleted and replaced by tablet b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:11.899189 1163 raft_consensus.cc:357] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.899304 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_be980d2eeeda473dbcfc4d7f282e21a9" } new_table_name: "default.Table_48c1aaaa0128418cb49437cb4d940dda" modify_external_catalogs: true
I20250625 07:52:11.899382 1163 raft_consensus.cc:383] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:11.899449 1163 raft_consensus.cc:738] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.899755 1163 consensus_queue.cc:260] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.899834 1158 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:11.900604 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8ac4c77f6adc4d03ae2404d42ae5a6d6" } new_table_name: "default.Table_e0125dcbb680481899e4adad86d7c2b5" modify_external_catalogs: true
I20250625 07:52:11.901943 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_90e4ef7124d54235bdea0b97ddaf7ecd" } new_table_name: "default.Table_dfdde9ef755044f7b3d9ab934ca4b231" modify_external_catalogs: true
I20250625 07:52:11.903919 1163 ts_tablet_manager.cc:1428] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250625 07:52:11.905782 1160 tablet_replica.cc:331] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:11.906541 1160 raft_consensus.cc:2241] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:11.905835 1163 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:11.908100 1162 raft_consensus.cc:357] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:11.908553 1162 raft_consensus.cc:383] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:11.908720 1162 raft_consensus.cc:738] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.909448 1163 tablet_bootstrap.cc:654] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:11.912454 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e252f6996cd42ccaa4054f0e16fcfa5 already in progress: creating tablet
W20250625 07:52:11.912736 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 because the tablet was not found. No further retry: Not found: Tablet not found: 753404a0a77c46e6a90fbacb6cdfdbe1
I20250625 07:52:11.913679 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.909533 1162 consensus_queue.cc:260] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:11.920326 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 9916612cb0234ec3a6df8f22c2b50e7c (table default.Table_9fdc22d5be4b43a4be699388015b9864 [id=7fbebf8268534597bd15d2bc70204cb7]). Expected version 1 got 0
I20250625 07:52:11.920722 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (table default.Table_be980d2eeeda473dbcfc4d7f282e21a9 [id=15c681d0005742e781fe94d2c8adc9d9]). Expected version 1 got 0
I20250625 07:52:11.921144 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet e90434606299495081d752df4761f9f8 (table default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]). Expected version 1 got 0
I20250625 07:52:11.921444 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet be1f3fbffc954fdb8eb5105b5d383d0c (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]). Expected version 1 got 0
I20250625 07:52:11.922655 1160 raft_consensus.cc:2270] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:11.923009 1162 ts_tablet_manager.cc:1428] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.035s user 0.016s sys 0.000s
I20250625 07:52:11.927042 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834d516547b046288e58b7cde4c56e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.927609 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51279fb42ae8480b8511432cb4f808ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.928881 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:11.931311 1160 ts_tablet_manager.cc:1905] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:11.933120 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4e5f06c153744fb7a5104d5e5ac47575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5f06c153744fb7a5104d5e5ac47575 already in progress: creating tablet
W20250625 07:52:11.933157 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4e5f06c153744fb7a5104d5e5ac47575 because the tablet was not found. No further retry: Not found: Tablet not found: 4e5f06c153744fb7a5104d5e5ac47575
W20250625 07:52:11.933466 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3471c83e015b44adbe8224c17a1c2eaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3471c83e015b44adbe8224c17a1c2eaf already in progress: creating tablet
W20250625 07:52:11.933722 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3e14b60ee7a843e5bb878148d8fd2e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e14b60ee7a843e5bb878148d8fd2e53 already in progress: creating tablet
W20250625 07:52:11.933780 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 688190c7d95541739902052507a884d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688190c7d95541739902052507a884d2 already in progress: creating tablet
W20250625 07:52:11.933969 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 95fce53f9a654fcaa8a52e5733c1b74f because the tablet was not found. No further retry: Not found: Tablet not found: 95fce53f9a654fcaa8a52e5733c1b74f
W20250625 07:52:11.933993 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753404a0a77c46e6a90fbacb6cdfdbe1 already in progress: creating tablet
W20250625 07:52:11.934156 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4e5f06c153744fb7a5104d5e5ac47575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5f06c153744fb7a5104d5e5ac47575 already in progress: creating tablet
W20250625 07:52:11.934182 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bf0189f6651347a18663a94aa61ee267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf0189f6651347a18663a94aa61ee267 already in progress: creating tablet
W20250625 07:52:11.934340 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3471c83e015b44adbe8224c17a1c2eaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3471c83e015b44adbe8224c17a1c2eaf already in progress: creating tablet
W20250625 07:52:11.934420 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162d6b0fc9e42f8b65446dc6efeb0b5 already in progress: creating tablet
W20250625 07:52:11.934523 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 already in progress: creating tablet
W20250625 07:52:11.934670 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 because the tablet was not found. No further retry: Not found: Tablet not found: 37f7c060fb8648b9aaa766bbaa1b4ba7
W20250625 07:52:11.934764 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3e14b60ee7a843e5bb878148d8fd2e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e14b60ee7a843e5bb878148d8fd2e53 already in progress: creating tablet
W20250625 07:52:11.934962 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 95fce53f9a654fcaa8a52e5733c1b74f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95fce53f9a654fcaa8a52e5733c1b74f already in progress: creating tablet
W20250625 07:52:11.935153 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bf0189f6651347a18663a94aa61ee267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf0189f6651347a18663a94aa61ee267 already in progress: creating tablet
W20250625 07:52:11.935321 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162d6b0fc9e42f8b65446dc6efeb0b5 already in progress: creating tablet
W20250625 07:52:11.935501 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a95c624abd8a45079c89fe1750faafcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95c624abd8a45079c89fe1750faafcf already in progress: creating tablet
I20250625 07:52:11.938045 1163 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:11.938218 1163 ts_tablet_manager.cc:1397] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.003s
I20250625 07:52:11.938982 1163 raft_consensus.cc:357] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.939186 1163 raft_consensus.cc:383] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:11.939291 1163 raft_consensus.cc:738] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.939512 1163 consensus_queue.cc:260] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.942039 1163 ts_tablet_manager.cc:1428] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:11.943101 1163 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:11.943603 1162 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:11.944849 1163 tablet_bootstrap.cc:654] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.946127 1162 tablet_bootstrap.cc:654] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.951695 1162 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:11.951936 1162 ts_tablet_manager.cc:1397] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20250625 07:52:11.953132 745 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.953358 1162 raft_consensus.cc:357] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.953620 1162 raft_consensus.cc:383] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:11.953735 1162 raft_consensus.cc:738] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.954039 1162 consensus_queue.cc:260] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:11.954591 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a95c624abd8a45079c89fe1750faafcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95c624abd8a45079c89fe1750faafcf already in progress: creating tablet
I20250625 07:52:11.956282 1158 tablet_bootstrap.cc:654] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.956560 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.956881 1166 tablet_replica.cc:331] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:11.957374 1166 raft_consensus.cc:2241] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:11.957870 1166 raft_consensus.cc:2270] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:11.958722 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 51279fb42ae8480b8511432cb4f808ab from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:11.959887 1166 ts_tablet_manager.cc:1905] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:11.961279 1162 ts_tablet_manager.cc:1428] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 07:52:11.961614 1162 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:11.963732 1162 tablet_bootstrap.cc:654] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:11.964175 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:11.966822 1158 log.cc:826] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Log is configured to *not* fsync() on all Append() calls
W20250625 07:52:11.970090 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94ef72e8a4f54fd4994dcf4725337478 consensus state has no leader
W20250625 07:52:11.970475 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 834d516547b046288e58b7cde4c56e5a consensus state has no leader
W20250625 07:52:11.970824 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 consensus state has no leader
I20250625 07:52:11.970973 1166 ts_tablet_manager.cc:1918] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:11.971114 1166 log.cc:1199] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/753404a0a77c46e6a90fbacb6cdfdbe1
W20250625 07:52:11.971176 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51279fb42ae8480b8511432cb4f808ab consensus state has no leader
W20250625 07:52:11.971532 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9861eea50d944b2c92be67c3c7c34582 consensus state has no leader
I20250625 07:52:11.971572 1166 ts_tablet_manager.cc:1939] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:11.977224 1166 tablet_replica.cc:331] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:11.978178 1166 raft_consensus.cc:2241] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:11.977969 1162 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:11.978436 1162 ts_tablet_manager.cc:1397] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250625 07:52:11.979607 1162 raft_consensus.cc:357] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.979856 1162 raft_consensus.cc:383] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:11.979986 1162 raft_consensus.cc:738] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:11.980293 1162 consensus_queue.cc:260] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:11.981755 1162 ts_tablet_manager.cc:1428] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:11.982031 1162 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:11.983711 1162 tablet_bootstrap.cc:654] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:11.985003 963 tablet.cc:2378] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:11.987211 1158 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:11.987560 1158 ts_tablet_manager.cc:1397] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.000s
I20250625 07:52:11.983913 1166 raft_consensus.cc:2270] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:11.989312 1166 ts_tablet_manager.cc:1905] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:11.993153 895 tablet_service.cc:1468] Processing CreateTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb (DEFAULT_TABLE table=default.Table_70696cfa27174f038c5b358f0684dd61 [id=516fdfcb90fd42d690397c77e99b4a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.993654 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d190e9d11c4589a82fcc7307e9eccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.993738 886 tablet_service.cc:1468] Processing CreateTablet for tablet b09dc0b8a496402b82f98daed9af0d12 (DEFAULT_TABLE table=default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf [id=7f77bfd2215a491d8fba698477109409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.994078 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09dc0b8a496402b82f98daed9af0d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.994951 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 (DEFAULT_TABLE table=default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.995818 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5b724ca63c436ba1ee776b8df3cec9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.000402 597 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet be1f3fbffc954fdb8eb5105b5d383d0c (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet be1f3fbffc954fdb8eb5105b5d383d0c consensus state has no leader
I20250625 07:52:11.993048 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b0d31993b4394cfd8832c93ab31b227b (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.993043 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.002140 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.002308 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.002485 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc7c0feb0e04ef28c323323e3007efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.002494 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d31993b4394cfd8832c93ab31b227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.002885 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.003722 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.006253 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc7c0feb0e04ef28c323323e3007efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.006361 893 tablet_service.cc:1468] Processing CreateTablet for tablet b0d31993b4394cfd8832c93ab31b227b (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.006705 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d31993b4394cfd8832c93ab31b227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.008052 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 51279fb42ae8480b8511432cb4f808ab
I20250625 07:52:12.008478 591 catalog_manager.cc:6652] ReplaceTablet: tablet 51279fb42ae8480b8511432cb4f808ab deleted and replaced by tablet 239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:12.008761 885 tablet_service.cc:1468] Processing CreateTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 (DEFAULT_TABLE table=default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.009101 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de10cbe72c8477ebe83831c869c3df1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.009533 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a95c624abd8a45079c89fe1750faafcf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95c624abd8a45079c89fe1750faafcf already in progress: creating tablet
W20250625 07:52:12.009776 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e252f6996cd42ccaa4054f0e16fcfa5 already in progress: creating tablet
I20250625 07:52:12.010071 1166 ts_tablet_manager.cc:1918] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.010208 1166 log.cc:1199] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/688190c7d95541739902052507a884d2
I20250625 07:52:12.010586 1166 ts_tablet_manager.cc:1939] T 688190c7d95541739902052507a884d2 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.010775 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.993307 1024 tablet_service.cc:1468] Processing CreateTablet for tablet a558e0a5403d4489892dfce9587e090c (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.010303 891 tablet_service.cc:1468] Processing CreateTablet for tablet a558e0a5403d4489892dfce9587e090c (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.013048 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a558e0a5403d4489892dfce9587e090c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.015710 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 51279fb42ae8480b8511432cb4f808ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51279fb42ae8480b8511432cb4f808ab already in progress: creating tablet
I20250625 07:52:12.015759 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a558e0a5403d4489892dfce9587e090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:11.993660 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 (DEFAULT_TABLE table=default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.999081 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.999320 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:11.999434 1026 tablet_service.cc:1468] Processing CreateTablet for tablet b09dc0b8a496402b82f98daed9af0d12 (DEFAULT_TABLE table=default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf [id=7f77bfd2215a491d8fba698477109409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:11.999853 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 (DEFAULT_TABLE table=default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.000118 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.000231 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.000751 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.001246 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb (DEFAULT_TABLE table=default.Table_70696cfa27174f038c5b358f0684dd61 [id=516fdfcb90fd42d690397c77e99b4a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.001344 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.001541 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.002000 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.018754 757 tablet_service.cc:1468] Processing CreateTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb (DEFAULT_TABLE table=default.Table_70696cfa27174f038c5b358f0684dd61 [id=516fdfcb90fd42d690397c77e99b4a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.018906 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de10cbe72c8477ebe83831c869c3df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.019276 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d190e9d11c4589a82fcc7307e9eccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.019918 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_48c1aaaa0128418cb49437cb4d940dda" } modify_external_catalogs: true
I20250625 07:52:12.019975 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_90101094ccf24d36a5fbb85a32358074" } modify_external_catalogs: true
I20250625 07:52:12.020166 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_48c1aaaa0128418cb49437cb4d940dda" } modify_external_catalogs: true
I20250625 07:52:12.020216 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_90101094ccf24d36a5fbb85a32358074" } modify_external_catalogs: true
I20250625 07:52:12.020591 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09dc0b8a496402b82f98daed9af0d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.020835 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5b724ca63c436ba1ee776b8df3cec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.020802 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1af8e3ae3fb04f159b97af62f5256fa8" } modify_external_catalogs: true
I20250625 07:52:12.020963 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1af8e3ae3fb04f159b97af62f5256fa8" } modify_external_catalogs: true
I20250625 07:52:12.021052 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e62955ce44346d290ac1a75c5daa6b6" } modify_external_catalogs: true
I20250625 07:52:12.021284 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d190e9d11c4589a82fcc7307e9eccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.021310 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e62955ce44346d290ac1a75c5daa6b6" } modify_external_catalogs: true
I20250625 07:52:12.022431 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dfdde9ef755044f7b3d9ab934ca4b231" } modify_external_catalogs: true
I20250625 07:52:12.022634 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dfdde9ef755044f7b3d9ab934ca4b231" } modify_external_catalogs: true
W20250625 07:52:12.022993 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3471c83e015b44adbe8224c17a1c2eaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3471c83e015b44adbe8224c17a1c2eaf already in progress: creating tablet
W20250625 07:52:12.023259 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4e5f06c153744fb7a5104d5e5ac47575 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5f06c153744fb7a5104d5e5ac47575 already in progress: creating tablet
W20250625 07:52:12.023654 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162d6b0fc9e42f8b65446dc6efeb0b5 already in progress: creating tablet
W20250625 07:52:12.028937 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 already in progress: creating tablet
W20250625 07:52:12.029250 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753404a0a77c46e6a90fbacb6cdfdbe1 already in progress: creating tablet
W20250625 07:52:12.029525 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 688190c7d95541739902052507a884d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688190c7d95541739902052507a884d2 already in progress: creating tablet
W20250625 07:52:12.029825 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bf0189f6651347a18663a94aa61ee267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf0189f6651347a18663a94aa61ee267 already in progress: creating tablet
I20250625 07:52:12.030709 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.031157 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc7c0feb0e04ef28c323323e3007efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.032199 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:12.038604 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3e14b60ee7a843e5bb878148d8fd2e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e14b60ee7a843e5bb878148d8fd2e53 already in progress: creating tablet
I20250625 07:52:12.039768 745 tablet_service.cc:1468] Processing CreateTablet for tablet b09dc0b8a496402b82f98daed9af0d12 (DEFAULT_TABLE table=default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf [id=7f77bfd2215a491d8fba698477109409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.040160 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b09dc0b8a496402b82f98daed9af0d12. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.042652 826 tablet.cc:2378] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:12.046298 1162 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.046478 1162 ts_tablet_manager.cc:1397] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.001s
I20250625 07:52:12.047492 745 tablet_service.cc:1468] Processing CreateTablet for tablet a558e0a5403d4489892dfce9587e090c (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.047721 1162 raft_consensus.cc:357] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.047930 1162 raft_consensus.cc:383] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.048012 1162 raft_consensus.cc:738] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.048063 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a558e0a5403d4489892dfce9587e090c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.048283 1162 consensus_queue.cc:260] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.048808 1162 ts_tablet_manager.cc:1428] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:12.049674 1158 raft_consensus.cc:357] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.050144 1158 raft_consensus.cc:383] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.050308 1158 raft_consensus.cc:738] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.051117 1158 consensus_queue.cc:260] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.053018 1158 ts_tablet_manager.cc:1428] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.051s user 0.010s sys 0.001s
I20250625 07:52:12.055510 1158 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.058024 1158 tablet_bootstrap.cc:654] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.060926 1162 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.063920 1162 tablet_bootstrap.cc:654] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.065116 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 (DEFAULT_TABLE table=default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.065284 755 tablet_service.cc:1468] Processing CreateTablet for tablet b0d31993b4394cfd8832c93ab31b227b (DEFAULT_TABLE table=default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.065666 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de10cbe72c8477ebe83831c869c3df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.066075 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d31993b4394cfd8832c93ab31b227b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.066319 1158 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.066478 1158 ts_tablet_manager.cc:1397] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250625 07:52:12.066075 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.067590 1166 tablet_replica.cc:331] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.067763 1166 raft_consensus.cc:2241] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.067945 1166 raft_consensus.cc:2270] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.068540 1166 ts_tablet_manager.cc:1905] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.069326 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.069620 1158 raft_consensus.cc:357] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.069741 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (table default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]) successfully deleted
I20250625 07:52:12.069821 1158 raft_consensus.cc:383] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.069893 1158 raft_consensus.cc:738] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.069985 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 688190c7d95541739902052507a884d2 (table default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]) successfully deleted
I20250625 07:52:12.070178 1158 consensus_queue.cc:260] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.070704 1158 ts_tablet_manager.cc:1428] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 07:52:12.070706 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.072489 1158 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.073782 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 753404a0a77c46e6a90fbacb6cdfdbe1 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.075047 1158 tablet_bootstrap.cc:654] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.077109 752 tablet_service.cc:1468] Processing CreateTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 (DEFAULT_TABLE table=default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.077522 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5b724ca63c436ba1ee776b8df3cec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.086627 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c44b0ef2ac60457bb56abead72f0182a (DEFAULT_TABLE table=default.Table_1a5bdf9178904b3a9d10e1621fb031e5 [id=f87841f2973841248d450e263c7a5d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.087091 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44b0ef2ac60457bb56abead72f0182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.092844 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.094591 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 572b063d66434e4796f6b86f8221e3c9 (DEFAULT_TABLE table=default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.095017 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572b063d66434e4796f6b86f8221e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.102149 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1222f2c1027b42c6ad9627359a93d678 (DEFAULT_TABLE table=default.Table_1041c1357c9e43e296c85b90037c3964 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.102566 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1222f2c1027b42c6ad9627359a93d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.102804 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d51a898202154881ab4ad41da1d519a3 (DEFAULT_TABLE table=default.Table_0c90f8a308d4444e8b015f34dd0c4140 [id=21758be8aff9485b9313da6b33405f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.103102 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51a898202154881ab4ad41da1d519a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.107061 750 tablet_service.cc:1468] Processing CreateTablet for tablet c44b0ef2ac60457bb56abead72f0182a (DEFAULT_TABLE table=default.Table_1a5bdf9178904b3a9d10e1621fb031e5 [id=f87841f2973841248d450e263c7a5d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.107507 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44b0ef2ac60457bb56abead72f0182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.111693 758 tablet_service.cc:1468] Processing CreateTablet for tablet 572b063d66434e4796f6b86f8221e3c9 (DEFAULT_TABLE table=default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.112044 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572b063d66434e4796f6b86f8221e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.114614 749 tablet_service.cc:1468] Processing CreateTablet for tablet d51a898202154881ab4ad41da1d519a3 (DEFAULT_TABLE table=default.Table_0c90f8a308d4444e8b015f34dd0c4140 [id=21758be8aff9485b9313da6b33405f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.115059 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51a898202154881ab4ad41da1d519a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.116755 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:12.117022 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1222f2c1027b42c6ad9627359a93d678 (DEFAULT_TABLE table=default.Table_1041c1357c9e43e296c85b90037c3964 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.117417 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1222f2c1027b42c6ad9627359a93d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.117553 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9861eea50d944b2c92be67c3c7c34582
I20250625 07:52:12.119410 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.120047 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.120167 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet be1f3fbffc954fdb8eb5105b5d383d0c
W20250625 07:52:12.120810 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5de10cbe72c8477ebe83831c869c3df1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de10cbe72c8477ebe83831c869c3df1 already in progress: creating tablet
W20250625 07:52:12.121039 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9861eea50d944b2c92be67c3c7c34582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861eea50d944b2c92be67c3c7c34582 already in progress: creating tablet
I20250625 07:52:12.121905 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.122191 1161 tablet_replica.cc:331] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.122712 1161 raft_consensus.cc:2241] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.123123 1161 raft_consensus.cc:2270] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.123464 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.123999 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.124851 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.125200 746 tablet_service.cc:1515] Processing DeleteTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.125304 1161 ts_tablet_manager.cc:1905] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.126487 884 tablet_service.cc:1468] Processing CreateTablet for tablet d51a898202154881ab4ad41da1d519a3 (DEFAULT_TABLE table=default.Table_0c90f8a308d4444e8b015f34dd0c4140 [id=21758be8aff9485b9313da6b33405f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.126924 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51a898202154881ab4ad41da1d519a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.128983 890 tablet_service.cc:1468] Processing CreateTablet for tablet 1222f2c1027b42c6ad9627359a93d678 (DEFAULT_TABLE table=default.Table_1041c1357c9e43e296c85b90037c3964 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.129354 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1222f2c1027b42c6ad9627359a93d678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.131060 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3ed8b54f47094fea9a6e3de748e5ca57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.131167 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_299334c718f547068b2ad26626e7f471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.131831 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_299334c718f547068b2ad26626e7f471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:12.131994 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.132190 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.132429 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.132683 883 tablet_service.cc:1515] Processing DeleteTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.133181 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_978181531aa34778bd274293434b1ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.133705 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_978181531aa34778bd274293434b1ac4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.134227 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.134789 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_20ee9c20a0714e32b1a10f14a48df181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.135211 1161 ts_tablet_manager.cc:1918] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:12.135236 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20ee9c20a0714e32b1a10f14a48df181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.135355 1161 log.cc:1199] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4e5f06c153744fb7a5104d5e5ac47575
I20250625 07:52:12.135748 1161 ts_tablet_manager.cc:1939] T 4e5f06c153744fb7a5104d5e5ac47575 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.135980 889 tablet_service.cc:1468] Processing CreateTablet for tablet 572b063d66434e4796f6b86f8221e3c9 (DEFAULT_TABLE table=default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.136349 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 572b063d66434e4796f6b86f8221e3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.136296 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7494d04fb7ec43c7929efbdf49d03103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.136749 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7494d04fb7ec43c7929efbdf49d03103 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.137809 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0d58b443a7b2466488c13f5ec50e1a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.138214 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d58b443a7b2466488c13f5ec50e1a0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:12.138597 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet be1f3fbffc954fdb8eb5105b5d383d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1f3fbffc954fdb8eb5105b5d383d0c already in progress: creating tablet
I20250625 07:52:12.138743 1162 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.138875 1161 tablet_replica.cc:331] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.138890 1162 ts_tablet_manager.cc:1397] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.001s
I20250625 07:52:12.139045 1161 raft_consensus.cc:2241] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.139194 1161 raft_consensus.cc:2270] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.139755 1162 raft_consensus.cc:357] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.139886 1161 ts_tablet_manager.cc:1905] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.139945 1162 raft_consensus.cc:383] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.140033 1162 raft_consensus.cc:738] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.140468 1162 consensus_queue.cc:260] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.140878 1158 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.141007 1158 ts_tablet_manager.cc:1397] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.004s
I20250625 07:52:12.142064 1158 raft_consensus.cc:357] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.142172 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8eab5a30488e4505bfd1a2a7149fe51d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.142302 1158 raft_consensus.cc:383] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.142501 1158 raft_consensus.cc:738] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:12.142656 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8eab5a30488e4505bfd1a2a7149fe51d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.142750 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0299e3ee1ac47be8d61de35bf8d8fee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.142827 1158 consensus_queue.cc:260] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:12.143323 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0299e3ee1ac47be8d61de35bf8d8fee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.145818 1158 ts_tablet_manager.cc:1428] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:12.147744 1158 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.149281 1162 ts_tablet_manager.cc:1428] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:12.149830 1158 tablet_bootstrap.cc:654] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.151562 1162 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.157450 1161 ts_tablet_manager.cc:1918] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.158262 886 tablet_service.cc:1468] Processing CreateTablet for tablet c44b0ef2ac60457bb56abead72f0182a (DEFAULT_TABLE table=default.Table_1a5bdf9178904b3a9d10e1621fb031e5 [id=f87841f2973841248d450e263c7a5d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.158923 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44b0ef2ac60457bb56abead72f0182a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.160159 1161 log.cc:1199] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1e252f6996cd42ccaa4054f0e16fcfa5
I20250625 07:52:12.160858 1161 ts_tablet_manager.cc:1939] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.162411 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (table default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]) successfully deleted
I20250625 07:52:12.170879 1166 ts_tablet_manager.cc:1918] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.171025 1166 log.cc:1199] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a95c624abd8a45079c89fe1750faafcf
I20250625 07:52:12.171380 1166 ts_tablet_manager.cc:1939] T a95c624abd8a45079c89fe1750faafcf P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.177006 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a95c624abd8a45079c89fe1750faafcf (table default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:12.181455 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:12.182520 1163 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.182758 1163 ts_tablet_manager.cc:1397] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.240s user 0.026s sys 0.053s
I20250625 07:52:12.183568 1163 raft_consensus.cc:357] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.184208 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.185075 1163 raft_consensus.cc:383] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.185248 1163 raft_consensus.cc:738] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.185583 1163 consensus_queue.cc:260] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.186164 1163 ts_tablet_manager.cc:1428] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:12.186398 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.186508 1163 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.186713 1161 tablet_replica.cc:331] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.186882 1161 raft_consensus.cc:2241] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.187062 1161 raft_consensus.cc:2270] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.187297 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.187721 1161 ts_tablet_manager.cc:1905] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:12.187844 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.188212 1163 tablet_bootstrap.cc:654] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.189146 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.189473 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.191649 1162 tablet_bootstrap.cc:654] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.197921 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.198311 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f774b4ac0c774ee7bf33d13a678cd368
W20250625 07:52:12.200748 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 51279fb42ae8480b8511432cb4f808ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51279fb42ae8480b8511432cb4f808ab already in progress: creating tablet
I20250625 07:52:12.201141 1166 tablet_replica.cc:331] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.201449 1166 raft_consensus.cc:2241] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.201648 1166 raft_consensus.cc:2270] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.202739 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.204437 1161 ts_tablet_manager.cc:1918] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.206032 1161 log.cc:1199] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e162d6b0fc9e42f8b65446dc6efeb0b5
I20250625 07:52:12.206404 752 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.211814 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 1222f2c1027b42c6ad9627359a93d678 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.212182 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dafc2205ebc14bd6b33cd83bf74099cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.212709 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.213670 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_70696cfa27174f038c5b358f0684dd61" } new_table_name: "default.Table_2d6ea477268d47f58b30102e5dc88635" modify_external_catalogs: true
I20250625 07:52:12.219194 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2cba52698a89454d9ea17c33d2d93fc3" } new_table_name: "default.Table_339dbe043bad47148d466ca915d84d33" modify_external_catalogs: true
I20250625 07:52:12.220218 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c90f8a308d4444e8b015f34dd0c4140" } new_table_name: "default.Table_9e0159dbcce04493a7cc2849bde83764" modify_external_catalogs: true
I20250625 07:52:12.223465 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1d369f97c52447bf90aa1a6f964a5199" } new_table_name: "default.Table_7ab4243dccfa40bda777a9a70016ec4f" modify_external_catalogs: true
W20250625 07:52:12.224820 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dafc2205ebc14bd6b33cd83bf74099cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.230746 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1041c1357c9e43e296c85b90037c3964" } new_table_name: "default.Table_ea8e512cd60044b791fb0d82facb1802" modify_external_catalogs: true
I20250625 07:52:12.238845 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 3e14b60ee7a843e5bb878148d8fd2e53 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.240502 589 master_service.cc:948] ReplaceTablet: received request to replace tablet a558e0a5403d4489892dfce9587e090c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.206214 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.241299 1161 ts_tablet_manager.cc:1939] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.241878 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (table default.Table_cea9385219bd4aadaa7ea96f51c8bfee [id=65a6d16ff5e444e286e0ca97322594c8]) successfully deleted
W20250625 07:52:12.242542 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 already in progress: creating tablet
W20250625 07:52:12.243078 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f774b4ac0c774ee7bf33d13a678cd368 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f774b4ac0c774ee7bf33d13a678cd368 already in progress: creating tablet
I20250625 07:52:12.243780 1166 ts_tablet_manager.cc:1905] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.246923 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 5de10cbe72c8477ebe83831c869c3df1 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.253700 1160 ts_tablet_manager.cc:1918] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.253922 1160 log.cc:1199] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3471c83e015b44adbe8224c17a1c2eaf
I20250625 07:52:12.254487 1160 ts_tablet_manager.cc:1939] T 3471c83e015b44adbe8224c17a1c2eaf P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.256657 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3471c83e015b44adbe8224c17a1c2eaf (table default.Table_1d369f97c52447bf90aa1a6f964a5199 [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:12.257723 1158 tablet_bootstrap.cc:492] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.257933 1158 ts_tablet_manager.cc:1397] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.022s
I20250625 07:52:12.259459 1158 raft_consensus.cc:357] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.259693 1158 raft_consensus.cc:383] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.259780 1158 raft_consensus.cc:738] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.260110 1158 consensus_queue.cc:260] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.265177 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:12.266469 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 688190c7d95541739902052507a884d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688190c7d95541739902052507a884d2 already in progress: creating tablet
I20250625 07:52:12.266765 1158 ts_tablet_manager.cc:1428] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
W20250625 07:52:12.268325 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55d190e9d11c4589a82fcc7307e9eccb consensus state has no leader
I20250625 07:52:12.269135 1158 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.269762 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 239e99c6b91d41d68a213a4c8d79470f (DEFAULT_TABLE table=default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.269939 886 tablet_service.cc:1468] Processing CreateTablet for tablet 239e99c6b91d41d68a213a4c8d79470f (DEFAULT_TABLE table=default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.270155 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239e99c6b91d41d68a213a4c8d79470f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.270331 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239e99c6b91d41d68a213a4c8d79470f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.276218 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 688190c7d95541739902052507a884d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.276736 1161 tablet_replica.cc:331] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.277055 1161 raft_consensus.cc:2241] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.277333 1161 raft_consensus.cc:2270] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.281790 752 tablet_service.cc:1468] Processing CreateTablet for tablet 239e99c6b91d41d68a213a4c8d79470f (DEFAULT_TABLE table=default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.282274 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239e99c6b91d41d68a213a4c8d79470f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.285629 1161 ts_tablet_manager.cc:1905] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.289467 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 94ef72e8a4f54fd4994dcf4725337478 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:12.290066 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94ef72e8a4f54fd4994dcf4725337478 consensus state has no leader
W20250625 07:52:12.291045 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d51a898202154881ab4ad41da1d519a3 consensus state has no leader
W20250625 07:52:12.296700 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 834d516547b046288e58b7cde4c56e5a consensus state has no leader
W20250625 07:52:12.298153 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3e14b60ee7a843e5bb878148d8fd2e53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e14b60ee7a843e5bb878148d8fd2e53 already in progress: creating tablet
W20250625 07:52:12.298413 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 95fce53f9a654fcaa8a52e5733c1b74f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95fce53f9a654fcaa8a52e5733c1b74f already in progress: creating tablet
W20250625 07:52:12.299160 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 already in progress: creating tablet
W20250625 07:52:12.300060 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e162d6b0fc9e42f8b65446dc6efeb0b5 already in progress: creating tablet
W20250625 07:52:12.300915 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bf0189f6651347a18663a94aa61ee267 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf0189f6651347a18663a94aa61ee267 already in progress: creating tablet
I20250625 07:52:12.301412 1160 tablet_replica.cc:331] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.302158 1158 tablet_bootstrap.cc:654] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:12.306016 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1222f2c1027b42c6ad9627359a93d678 consensus state has no leader
I20250625 07:52:12.311036 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 (DEFAULT_TABLE table=default.Table_e0299e3ee1ac47be8d61de35bf8d8fee [id=02ec3b2efaed40e7b6354e1f15d78043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.311591 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3488a9fb19452dac3241170dd2fee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.311879 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 (DEFAULT_TABLE table=default.Table_e0299e3ee1ac47be8d61de35bf8d8fee [id=02ec3b2efaed40e7b6354e1f15d78043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.312306 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3488a9fb19452dac3241170dd2fee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.312575 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa (DEFAULT_TABLE table=default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.312937 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b020ff9f04b4d1796af4ff585cc04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.313694 1161 ts_tablet_manager.cc:1918] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.313848 1161 log.cc:1199] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/688190c7d95541739902052507a884d2
I20250625 07:52:12.314240 1161 ts_tablet_manager.cc:1939] T 688190c7d95541739902052507a884d2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.315317 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 688190c7d95541739902052507a884d2 (table default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]) successfully deleted
I20250625 07:52:12.302337 1160 raft_consensus.cc:2241] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.311995 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 (DEFAULT_TABLE table=default.Table_e0299e3ee1ac47be8d61de35bf8d8fee [id=02ec3b2efaed40e7b6354e1f15d78043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.317783 1158 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.317894 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3488a9fb19452dac3241170dd2fee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.317942 1158 ts_tablet_manager.cc:1397] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250625 07:52:12.318789 1158 raft_consensus.cc:357] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:12.318965 1158 raft_consensus.cc:383] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.319056 1158 raft_consensus.cc:738] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.319267 1158 consensus_queue.cc:260] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:12.319824 1158 ts_tablet_manager.cc:1428] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:12.320091 1158 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.326409 752 tablet_service.cc:1468] Processing CreateTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa (DEFAULT_TABLE table=default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.326864 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b020ff9f04b4d1796af4ff585cc04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.328794 1160 raft_consensus.cc:2270] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.331657 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8eab5a30488e4505bfd1a2a7149fe51d" } modify_external_catalogs: true
I20250625 07:52:12.331931 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8eab5a30488e4505bfd1a2a7149fe51d" } modify_external_catalogs: true
I20250625 07:52:12.332861 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.333755 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dafc2205ebc14bd6b33cd83bf74099cf" } modify_external_catalogs: true
I20250625 07:52:12.333981 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dafc2205ebc14bd6b33cd83bf74099cf" } modify_external_catalogs: true
I20250625 07:52:12.334875 1162 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.335044 1162 ts_tablet_manager.cc:1397] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.184s user 0.006s sys 0.016s
I20250625 07:52:12.335626 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_601b836403094b4cb7106f74b03172e7" } modify_external_catalogs: true
I20250625 07:52:12.335808 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_601b836403094b4cb7106f74b03172e7" } modify_external_catalogs: true
I20250625 07:52:12.335806 1162 raft_consensus.cc:357] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.337113 1162 raft_consensus.cc:383] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.337209 1162 raft_consensus.cc:738] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.337795 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu pid 424
I20250625 07:52:12.339015 1160 ts_tablet_manager.cc:1905] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.339308 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 07:52:12.340875 1162 consensus_queue.cc:260] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.341019 749 tablet_service.cc:1468] Processing CreateTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 (DEFAULT_TABLE table=default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.341444 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b3830bd506443ab6f06ddd3c6b9974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.341094 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa (DEFAULT_TABLE table=default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.343148 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b020ff9f04b4d1796af4ff585cc04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.345685 1162 ts_tablet_manager.cc:1428] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250625 07:52:12.348565 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.348865 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:12.349407 592 catalog_manager.cc:6652] ReplaceTablet: tablet 1222f2c1027b42c6ad9627359a93d678 deleted and replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:12.349592 1162 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.349679 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.350760 1158 tablet_bootstrap.cc:654] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.351081 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.353749 1162 tablet_bootstrap.cc:654] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.354076 1160 ts_tablet_manager.cc:1918] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.354285 1160 log.cc:1199] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a95c624abd8a45079c89fe1750faafcf
I20250625 07:52:12.354719 1160 ts_tablet_manager.cc:1939] T a95c624abd8a45079c89fe1750faafcf P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.355327 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:12.355500 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a558e0a5403d4489892dfce9587e090c
I20250625 07:52:12.355826 595 catalog_manager.cc:6652] ReplaceTablet: tablet 94ef72e8a4f54fd4994dcf4725337478 deleted and replaced by tablet 365ef91e7bac43e283602d3d036853ff
I20250625 07:52:12.355881 589 catalog_manager.cc:6652] ReplaceTablet: tablet a558e0a5403d4489892dfce9587e090c deleted and replaced by tablet 1eab049ad7fc4b5581e5c31a14491306
I20250625 07:52:12.355979 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a95c624abd8a45079c89fe1750faafcf (table default.Table_d060de5813af448fa1de9410a0d6e4cd [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:12.356190 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3338d1c708b4421c8501740da89afec5
I20250625 07:52:12.356931 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.356941 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.357080 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.357328 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.358002 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.358055 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 (DEFAULT_TABLE table=default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.358263 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.358538 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.358608 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b3830bd506443ab6f06ddd3c6b9974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.358781 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.359105 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.359443 1160 tablet_replica.cc:331] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.359651 1160 raft_consensus.cc:2241] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.360031 1160 raft_consensus.cc:2270] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.360843 1160 ts_tablet_manager.cc:1905] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.362169 1158 tablet_bootstrap.cc:492] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.362290 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.362351 1158 ts_tablet_manager.cc:1397] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.004s
I20250625 07:52:12.363559 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:12.363682 594 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 3dd64c55167c423fa4408db65c353775
I20250625 07:52:12.363577 1158 raft_consensus.cc:357] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.363782 1158 raft_consensus.cc:383] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.363854 1158 raft_consensus.cc:738] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.364682 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.365804 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.366326 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.367698 749 tablet_service.cc:1468] Processing CreateTablet for tablet 012df7beedb1446ca5e533d6624afc10 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.368101 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012df7beedb1446ca5e533d6624afc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.368558 886 tablet_service.cc:1468] Processing CreateTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 (DEFAULT_TABLE table=default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.369005 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b3830bd506443ab6f06ddd3c6b9974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.369084 1158 consensus_queue.cc:260] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.373121 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 012df7beedb1446ca5e533d6624afc10 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.373265 1166 ts_tablet_manager.cc:1918] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.373592 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012df7beedb1446ca5e533d6624afc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.374596 1158 ts_tablet_manager.cc:1428] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:12.375273 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3338d1c708b4421c8501740da89afec5 (DEFAULT_TABLE table=default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.375677 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3338d1c708b4421c8501740da89afec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.376861 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3471c83e015b44adbe8224c17a1c2eaf with delete_type TABLET_DATA_DELETED (replaced by tablet 834d516547b046288e58b7cde4c56e5a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.379398 887 tablet_service.cc:1468] Processing CreateTablet for tablet 23755f70822740e08ca6a5e5b5571458 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.380470 889 tablet_service.cc:1468] Processing CreateTablet for tablet b7139335c7124ab79bc2d0a251984199 (DEFAULT_TABLE table=default.Table_299334c718f547068b2ad26626e7f471 [id=84244d54b35c47829f8f8a92c3d47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.380872 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3338d1c708b4421c8501740da89afec5 (DEFAULT_TABLE table=default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.381302 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23755f70822740e08ca6a5e5b5571458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.383579 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (DEFAULT_TABLE table=default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.384672 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_20ee9c20a0714e32b1a10f14a48df181" } modify_external_catalogs: true
I20250625 07:52:12.384766 885 tablet_service.cc:1468] Processing CreateTablet for tablet 86ea9df068a94d9187f2f9620e371a39 (DEFAULT_TABLE table=default.Table_7494d04fb7ec43c7929efbdf49d03103 [id=5281aca609cf421990f96de3424b4442]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.384903 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_20ee9c20a0714e32b1a10f14a48df181" } modify_external_catalogs: true
I20250625 07:52:12.385145 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea9df068a94d9187f2f9620e371a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.385267 750 tablet_service.cc:1468] Processing CreateTablet for tablet b7139335c7124ab79bc2d0a251984199 (DEFAULT_TABLE table=default.Table_299334c718f547068b2ad26626e7f471 [id=84244d54b35c47829f8f8a92c3d47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.385637 758 tablet_service.cc:1468] Processing CreateTablet for tablet 23755f70822740e08ca6a5e5b5571458 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.385965 755 tablet_service.cc:1468] Processing CreateTablet for tablet 86ea9df068a94d9187f2f9620e371a39 (DEFAULT_TABLE table=default.Table_7494d04fb7ec43c7929efbdf49d03103 [id=5281aca609cf421990f96de3424b4442]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.386286 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.387054 1024 tablet_service.cc:1468] Processing CreateTablet for tablet b7139335c7124ab79bc2d0a251984199 (DEFAULT_TABLE table=default.Table_299334c718f547068b2ad26626e7f471 [id=84244d54b35c47829f8f8a92c3d47ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.387601 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 23755f70822740e08ca6a5e5b5571458 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.388805 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea9df068a94d9187f2f9620e371a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.389405 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 86ea9df068a94d9187f2f9620e371a39 (DEFAULT_TABLE table=default.Table_7494d04fb7ec43c7929efbdf49d03103 [id=5281aca609cf421990f96de3424b4442]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.390004 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (DEFAULT_TABLE table=default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.390269 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23755f70822740e08ca6a5e5b5571458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.392571 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3338d1c708b4421c8501740da89afec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.392824 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7139335c7124ab79bc2d0a251984199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.396739 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7139335c7124ab79bc2d0a251984199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.396801 1162 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.396963 1162 ts_tablet_manager.cc:1397] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.004s
I20250625 07:52:12.398336 1162 raft_consensus.cc:357] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.398531 1162 raft_consensus.cc:383] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.398630 1162 raft_consensus.cc:738] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.398855 1162 consensus_queue.cc:260] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.399411 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3338d1c708b4421c8501740da89afec5 (DEFAULT_TABLE table=default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.399832 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3338d1c708b4421c8501740da89afec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.401774 1162 ts_tablet_manager.cc:1428] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 07:52:12.402079 1162 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.403860 1162 tablet_bootstrap.cc:654] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.405632 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2d6ea477268d47f58b30102e5dc88635" } modify_external_catalogs: true
I20250625 07:52:12.405830 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2d6ea477268d47f58b30102e5dc88635" } modify_external_catalogs: true
I20250625 07:52:12.407622 1166 log.cc:1199] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1e252f6996cd42ccaa4054f0e16fcfa5
I20250625 07:52:12.409106 1166 ts_tablet_manager.cc:1939] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.410315 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (table default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]) successfully deleted
I20250625 07:52:12.412973 884 tablet_service.cc:1468] Processing CreateTablet for tablet 012df7beedb1446ca5e533d6624afc10 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.413473 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012df7beedb1446ca5e533d6624afc10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.413590 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2b020ff9f04b4d1796af4ff585cc04aa
I20250625 07:52:12.414633 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.418689 1161 tablet_replica.cc:331] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.418783 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7139335c7124ab79bc2d0a251984199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.418895 1161 raft_consensus.cc:2241] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.419101 1161 raft_consensus.cc:2270] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.419301 1158 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.419512 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 365ef91e7bac43e283602d3d036853ff (DEFAULT_TABLE table=default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.419806 1161 ts_tablet_manager.cc:1905] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.420409 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365ef91e7bac43e283602d3d036853ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.421080 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23755f70822740e08ca6a5e5b5571458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.421814 1158 tablet_bootstrap.cc:654] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.421970 887 tablet_service.cc:1468] Processing CreateTablet for tablet 365ef91e7bac43e283602d3d036853ff (DEFAULT_TABLE table=default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.422399 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_98b635139b69481f882d18773584a8e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.423480 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:12.416831 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5de10cbe72c8477ebe83831c869c3df1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de10cbe72c8477ebe83831c869c3df1 already in progress: creating tablet
I20250625 07:52:12.424893 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_463abb0995a149ffb2544e9ab2110eaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.425990 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (DEFAULT_TABLE table=default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.426043 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4efdc728e9ca49549585528cc7eb22ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.426401 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.426733 1166 tablet_replica.cc:331] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.426471 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (DEFAULT_TABLE table=default.Table_ea8e512cd60044b791fb0d82facb1802 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:12.426824 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b635139b69481f882d18773584a8e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.426985 1166 raft_consensus.cc:2241] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.427191 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365ef91e7bac43e283602d3d036853ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.427464 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.427501 1166 raft_consensus.cc:2270] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.429337 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ac3fbcd8a67040aa82930017de78ea14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.429770 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac3fbcd8a67040aa82930017de78ea14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.430722 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1786d469d14e46baa8b15791a16c0456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.431138 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1786d469d14e46baa8b15791a16c0456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.431934 1166 ts_tablet_manager.cc:1905] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:12.432782 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_463abb0995a149ffb2544e9ab2110eaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.432832 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdb6a689ce6440ab7037484e2b9cdc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.432718 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b17157c58c6a40f7ad4c8dfaa52b410f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.433326 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17157c58c6a40f7ad4c8dfaa52b410f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:12.434312 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4efdc728e9ca49549585528cc7eb22ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:12.439355 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9861eea50d944b2c92be67c3c7c34582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861eea50d944b2c92be67c3c7c34582 already in progress: creating tablet
I20250625 07:52:12.440378 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5454db9f8f894091981c0f6abdfd2955"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.440960 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5454db9f8f894091981c0f6abdfd2955 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.441699 1163 tablet_bootstrap.cc:492] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.441848 1163 ts_tablet_manager.cc:1397] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.255s user 0.039s sys 0.054s
I20250625 07:52:12.442677 1163 raft_consensus.cc:357] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.442847 1163 raft_consensus.cc:383] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.442937 1163 raft_consensus.cc:738] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.443164 1163 consensus_queue.cc:260] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.443672 1163 ts_tablet_manager.cc:1428] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:12.443930 1163 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.444732 1160 ts_tablet_manager.cc:1918] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.444849 1160 log.cc:1199] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4e5f06c153744fb7a5104d5e5ac47575
I20250625 07:52:12.445461 1160 ts_tablet_manager.cc:1939] T 4e5f06c153744fb7a5104d5e5ac47575 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.447283 1160 tablet_replica.cc:331] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.447530 1160 raft_consensus.cc:2241] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.447686 1160 raft_consensus.cc:2270] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:12.448067 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 51279fb42ae8480b8511432cb4f808ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51279fb42ae8480b8511432cb4f808ab already in progress: creating tablet
W20250625 07:52:12.448282 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5de10cbe72c8477ebe83831c869c3df1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de10cbe72c8477ebe83831c869c3df1 already in progress: creating tablet
W20250625 07:52:12.448473 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9861eea50d944b2c92be67c3c7c34582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861eea50d944b2c92be67c3c7c34582 already in progress: creating tablet
W20250625 07:52:12.448632 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet be1f3fbffc954fdb8eb5105b5d383d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be1f3fbffc954fdb8eb5105b5d383d0c already in progress: creating tablet
W20250625 07:52:12.448855 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 already in progress: creating tablet
W20250625 07:52:12.449055 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f774b4ac0c774ee7bf33d13a678cd368 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f774b4ac0c774ee7bf33d13a678cd368 already in progress: creating tablet
I20250625 07:52:12.450377 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d6ac142574f64d839444fe2bf4673aed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.450958 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6ac142574f64d839444fe2bf4673aed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.451663 1163 tablet_bootstrap.cc:654] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.453049 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e252f6996cd42ccaa4054f0e16fcfa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.453348 1160 ts_tablet_manager.cc:1905] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.453922 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ea9df068a94d9187f2f9620e371a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.455371 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.455883 752 tablet_service.cc:1468] Processing CreateTablet for tablet 365ef91e7bac43e283602d3d036853ff (DEFAULT_TABLE table=default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.456357 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 365ef91e7bac43e283602d3d036853ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.460866 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 94ef72e8a4f54fd4994dcf4725337478 (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 94ef72e8a4f54fd4994dcf4725337478 consensus state has no leader
I20250625 07:52:12.461644 1166 ts_tablet_manager.cc:1918] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.461779 1166 log.cc:1199] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:12.462128 1166 ts_tablet_manager.cc:1939] T be1f3fbffc954fdb8eb5105b5d383d0c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.463181 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.463609 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eab049ad7fc4b5581e5c31a14491306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.465656 1160 ts_tablet_manager.cc:1918] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.465754 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (DEFAULT_TABLE table=default.Table_ea8e512cd60044b791fb0d82facb1802 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.466096 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdb6a689ce6440ab7037484e2b9cdc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.466271 1160 log.cc:1199] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/688190c7d95541739902052507a884d2
I20250625 07:52:12.467187 1160 ts_tablet_manager.cc:1939] T 688190c7d95541739902052507a884d2 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.468287 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 688190c7d95541739902052507a884d2 (table default.Table_99b37e062dfa4c5cb24ca5a7a1d70ae5 [id=7e0f89d5dd7e4ca09f68688f18685e13]) successfully deleted
W20250625 07:52:12.470970 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1222f2c1027b42c6ad9627359a93d678 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1222f2c1027b42c6ad9627359a93d678 already in progress: creating tablet
W20250625 07:52:12.471879 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 94ef72e8a4f54fd4994dcf4725337478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ef72e8a4f54fd4994dcf4725337478 already in progress: creating tablet
I20250625 07:52:12.472010 1166 tablet_replica.cc:331] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.472208 1166 raft_consensus.cc:2241] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.472397 1166 raft_consensus.cc:2270] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:12.472621 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3338d1c708b4421c8501740da89afec5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3338d1c708b4421c8501740da89afec5 already in progress: creating tablet
W20250625 07:52:12.473388 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a558e0a5403d4489892dfce9587e090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a558e0a5403d4489892dfce9587e090c already in progress: creating tablet
I20250625 07:52:12.473856 1166 ts_tablet_manager.cc:1905] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:12.474109 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 572b063d66434e4796f6b86f8221e3c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572b063d66434e4796f6b86f8221e3c9 already in progress: creating tablet
W20250625 07:52:12.474926 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2b020ff9f04b4d1796af4ff585cc04aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b020ff9f04b4d1796af4ff585cc04aa already in progress: creating tablet
I20250625 07:52:12.475462 1160 tablet_replica.cc:331] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.476034 1160 raft_consensus.cc:2241] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.476387 1160 raft_consensus.cc:2270] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.477350 1160 ts_tablet_manager.cc:1905] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.478065 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b354149a4d24041b4a1a28faa3782d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.478662 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b354149a4d24041b4a1a28faa3782d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.479898 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:12.480119 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 012df7beedb1446ca5e533d6624afc10 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.479903 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7494d04fb7ec43c7929efbdf49d03103" } new_table_name: "default.Table_17d5751097fa40feaf90d0fb3c97e1b5" modify_external_catalogs: true
I20250625 07:52:12.481385 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.482864 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet be1f3fbffc954fdb8eb5105b5d383d0c (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]) successfully deleted
W20250625 07:52:12.483225 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 already in progress: creating tablet
I20250625 07:52:12.486024 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:12.487021 1098 tablet.cc:2378] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:12.487785 1166 ts_tablet_manager.cc:1918] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.487922 1166 log.cc:1199] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3471c83e015b44adbe8224c17a1c2eaf
I20250625 07:52:12.488251 1166 ts_tablet_manager.cc:1939] T 3471c83e015b44adbe8224c17a1c2eaf P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.491125 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.493198 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1a5bdf9178904b3a9d10e1621fb031e5" } new_table_name: "default.Table_808e1666e3d347168df6c5a4301fc6d8" modify_external_catalogs: true
I20250625 07:52:12.494596 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_463abb0995a149ffb2544e9ab2110eaa" } new_table_name: "default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c" modify_external_catalogs: true
I20250625 07:52:12.495811 1166 tablet_replica.cc:331] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.496043 1166 raft_consensus.cc:2241] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.496240 1166 raft_consensus.cc:2270] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.498319 597 master_service.cc:948] ReplaceTablet: received request to replace tablet bf0189f6651347a18663a94aa61ee267 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.498360 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d060de5813af448fa1de9410a0d6e4cd" } new_table_name: "default.Table_a2747cfb800e440eb843d823139c4f27" modify_external_catalogs: true
I20250625 07:52:12.499552 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3471c83e015b44adbe8224c17a1c2eaf (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:12.499922 1160 ts_tablet_manager.cc:1918] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:12.499933 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f774b4ac0c774ee7bf33d13a678cd368 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f774b4ac0c774ee7bf33d13a678cd368 already in progress: creating tablet
I20250625 07:52:12.499949 1166 ts_tablet_manager.cc:1905] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.500586 1160 log.cc:1199] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1e252f6996cd42ccaa4054f0e16fcfa5
I20250625 07:52:12.501127 887 tablet_service.cc:1468] Processing CreateTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.501742 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eab049ad7fc4b5581e5c31a14491306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.502465 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ac4400688af8429ea02aeec0e8d6b03b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.502985 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4400688af8429ea02aeec0e8d6b03b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.507375 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b8fd98fa1d8340a9ab0f6ea95ae92cbf" } new_table_name: "default.Table_c931105d39a449fdbdc73dc16830a6a4" modify_external_catalogs: true
I20250625 07:52:12.507663 1162 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.507830 1162 ts_tablet_manager.cc:1397] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.039s
I20250625 07:52:12.508654 1162 raft_consensus.cc:357] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.508812 1160 ts_tablet_manager.cc:1939] T 1e252f6996cd42ccaa4054f0e16fcfa5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.509253 1162 raft_consensus.cc:383] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.509349 1162 raft_consensus.cc:738] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.509794 1162 consensus_queue.cc:260] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.510962 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1e252f6996cd42ccaa4054f0e16fcfa5 (table default.Table_22c7b46f7f5f4e9c8dd9f0c9dae5cfd9 [id=7f770289170342569d84d5ac6f9729d2]) successfully deleted
W20250625 07:52:12.512646 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 55d190e9d11c4589a82fcc7307e9eccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d190e9d11c4589a82fcc7307e9eccb already in progress: creating tablet
I20250625 07:52:12.513314 1162 ts_tablet_manager.cc:1428] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:12.493497 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (DEFAULT_TABLE table=default.Table_ea8e512cd60044b791fb0d82facb1802 [id=5174ca174ab24d7695eb1fddc9414c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.517019 1162 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.517433 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.519352 1162 tablet_bootstrap.cc:654] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.524035 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 012df7beedb1446ca5e533d6624afc10
I20250625 07:52:12.524531 590 catalog_manager.cc:6652] ReplaceTablet: tablet 012df7beedb1446ca5e533d6624afc10 deleted and replaced by tablet aec412a15b964060a24631e7ebcf1ef7
W20250625 07:52:12.524919 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86ea9df068a94d9187f2f9620e371a39 consensus state has no leader
W20250625 07:52:12.525338 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c44b0ef2ac60457bb56abead72f0182a consensus state has no leader
W20250625 07:52:12.525683 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b71aadf074a4a39943b2ff930909f93 has no consensus state
I20250625 07:52:12.526309 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.526532 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.528671 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdb6a689ce6440ab7037484e2b9cdc5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.529884 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d31993b4394cfd8832c93ab31b227b consensus state has no leader
I20250625 07:52:12.534519 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 (DEFAULT_TABLE table=default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:12.535006 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b09dc0b8a496402b82f98daed9af0d12 consensus state has no leader
W20250625 07:52:12.535555 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 012df7beedb1446ca5e533d6624afc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012df7beedb1446ca5e533d6624afc10 already in progress: creating tablet
I20250625 07:52:12.538311 1163 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.538605 1163 ts_tablet_manager.cc:1397] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.030s
I20250625 07:52:12.539673 1163 raft_consensus.cc:357] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.540001 1163 raft_consensus.cc:383] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.540108 1163 raft_consensus.cc:738] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.540405 1163 consensus_queue.cc:260] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.540621 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1eab049ad7fc4b5581e5c31a14491306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.540977 1163 ts_tablet_manager.cc:1428] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:12.541335 1163 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.534766 1161 ts_tablet_manager.cc:1918] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.542968 1161 log.cc:1199] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/753404a0a77c46e6a90fbacb6cdfdbe1
I20250625 07:52:12.543470 1161 ts_tablet_manager.cc:1939] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.545045 1161 tablet_replica.cc:331] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.545279 1161 raft_consensus.cc:2241] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.545497 1161 raft_consensus.cc:2270] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.546113 752 tablet_service.cc:1468] Processing CreateTablet for tablet 9e800d4982fa45f4b931985c8b52719d (DEFAULT_TABLE table=default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.546283 1161 ts_tablet_manager.cc:1905] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.546813 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e800d4982fa45f4b931985c8b52719d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.546787 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (table default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]) successfully deleted
I20250625 07:52:12.546914 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:12.547016 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1222f2c1027b42c6ad9627359a93d678 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1222f2c1027b42c6ad9627359a93d678 already in progress: creating tablet
W20250625 07:52:12.547266 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 94ef72e8a4f54fd4994dcf4725337478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ef72e8a4f54fd4994dcf4725337478 already in progress: creating tablet
W20250625 07:52:12.547995 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3338d1c708b4421c8501740da89afec5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3338d1c708b4421c8501740da89afec5 already in progress: creating tablet
W20250625 07:52:12.548231 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a558e0a5403d4489892dfce9587e090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a558e0a5403d4489892dfce9587e090c already in progress: creating tablet
W20250625 07:52:12.548410 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 572b063d66434e4796f6b86f8221e3c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572b063d66434e4796f6b86f8221e3c9 already in progress: creating tablet
I20250625 07:52:12.550146 893 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.550215 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6b71aadf074a4a39943b2ff930909f93 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.550611 1160 tablet_replica.cc:331] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.550558 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3eea1c257b584a029efa08922547b0cc (DEFAULT_TABLE table=default.Table_ac3fbcd8a67040aa82930017de78ea14 [id=3509d763da554384ba8b5ecefdbc5fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.550801 1160 raft_consensus.cc:2241] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.550968 1160 raft_consensus.cc:2270] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.551040 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a (DEFAULT_TABLE table=default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.551349 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b71aadf074a4a39943b2ff930909f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.551577 889 tablet_service.cc:1468] Processing CreateTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d (DEFAULT_TABLE table=default.Table_b17157c58c6a40f7ad4c8dfaa52b410f [id=975c0e95149a474b8e98ad4c4411b385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.551874 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81396ee6ed7400baa30dcc5d71ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.552088 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a (DEFAULT_TABLE table=default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.552536 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e8d65c2ed5240fc923a7c7b59488e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.552953 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5de1492e483e47919ee5581ecee227be (DEFAULT_TABLE table=default.Table_d6ac142574f64d839444fe2bf4673aed [id=23322aa9321140c39272b5b2a102fb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.553331 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de1492e483e47919ee5581ecee227be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.554075 751 tablet_service.cc:1468] Processing CreateTablet for tablet 3eea1c257b584a029efa08922547b0cc (DEFAULT_TABLE table=default.Table_ac3fbcd8a67040aa82930017de78ea14 [id=3509d763da554384ba8b5ecefdbc5fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.554440 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eea1c257b584a029efa08922547b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.556952 888 tablet_service.cc:1468] Processing CreateTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (DEFAULT_TABLE table=default.Table_1786d469d14e46baa8b15791a16c0456 [id=58c1b09d38bc456280bf241369bef00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.557286 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1770f6d5d1d4899a66ec23ab4eadc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.545024 887 tablet_service.cc:1468] Processing CreateTablet for tablet 32bd7cdef96946e88683e457c2f26c9b (DEFAULT_TABLE table=default.Table_4efdc728e9ca49549585528cc7eb22ee [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.549785 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9e800d4982fa45f4b931985c8b52719d (DEFAULT_TABLE table=default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.558780 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9fdc22d5be4b43a4be699388015b9864" } modify_external_catalogs: true
I20250625 07:52:12.559039 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9fdc22d5be4b43a4be699388015b9864" } modify_external_catalogs: true
I20250625 07:52:12.559626 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a (DEFAULT_TABLE table=default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.559834 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5de1492e483e47919ee5581ecee227be (DEFAULT_TABLE table=default.Table_d6ac142574f64d839444fe2bf4673aed [id=23322aa9321140c39272b5b2a102fb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.560470 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 6b71aadf074a4a39943b2ff930909f93 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.560638 1024 tablet_service.cc:1468] Processing CreateTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d (DEFAULT_TABLE table=default.Table_b17157c58c6a40f7ad4c8dfaa52b410f [id=975c0e95149a474b8e98ad4c4411b385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.561141 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (DEFAULT_TABLE table=default.Table_1786d469d14e46baa8b15791a16c0456 [id=58c1b09d38bc456280bf241369bef00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.561283 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9e800d4982fa45f4b931985c8b52719d (DEFAULT_TABLE table=default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.561954 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3eea1c257b584a029efa08922547b0cc (DEFAULT_TABLE table=default.Table_ac3fbcd8a67040aa82930017de78ea14 [id=3509d763da554384ba8b5ecefdbc5fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.565012 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9e0159dbcce04493a7cc2849bde83764" } modify_external_catalogs: true
I20250625 07:52:12.565205 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9e0159dbcce04493a7cc2849bde83764" } modify_external_catalogs: true
I20250625 07:52:12.566893 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5de1492e483e47919ee5581ecee227be (DEFAULT_TABLE table=default.Table_d6ac142574f64d839444fe2bf4673aed [id=23322aa9321140c39272b5b2a102fb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.567072 749 tablet_service.cc:1468] Processing CreateTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (DEFAULT_TABLE table=default.Table_1786d469d14e46baa8b15791a16c0456 [id=58c1b09d38bc456280bf241369bef00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.567267 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de1492e483e47919ee5581ecee227be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.567445 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1770f6d5d1d4899a66ec23ab4eadc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.567708 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eea1c257b584a029efa08922547b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.568316 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6b71aadf074a4a39943b2ff930909f93 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.568435 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e800d4982fa45f4b931985c8b52719d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.568663 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b71aadf074a4a39943b2ff930909f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.569772 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e8d65c2ed5240fc923a7c7b59488e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.570000 758 tablet_service.cc:1468] Processing CreateTablet for tablet 32bd7cdef96946e88683e457c2f26c9b (DEFAULT_TABLE table=default.Table_4efdc728e9ca49549585528cc7eb22ee [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.570376 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32bd7cdef96946e88683e457c2f26c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.572800 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32bd7cdef96946e88683e457c2f26c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.573160 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98b635139b69481f882d18773584a8e7" } modify_external_catalogs: true
I20250625 07:52:12.573362 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98b635139b69481f882d18773584a8e7" } modify_external_catalogs: true
I20250625 07:52:12.575827 1163 tablet_bootstrap.cc:654] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.576773 750 tablet_service.cc:1468] Processing CreateTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d (DEFAULT_TABLE table=default.Table_b17157c58c6a40f7ad4c8dfaa52b410f [id=975c0e95149a474b8e98ad4c4411b385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.578534 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81396ee6ed7400baa30dcc5d71ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.591035 1158 tablet_bootstrap.cc:492] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.591400 1158 ts_tablet_manager.cc:1397] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.214s user 0.004s sys 0.004s
I20250625 07:52:12.595690 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c (DEFAULT_TABLE table=default.Table_ac4400688af8429ea02aeec0e8d6b03b [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.596323 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c (DEFAULT_TABLE table=default.Table_ac4400688af8429ea02aeec0e8d6b03b [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.596726 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1eaf0266c04873b185c1c0c2da201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.597007 891 tablet_service.cc:1468] Processing CreateTablet for tablet aec412a15b964060a24631e7ebcf1ef7 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.595260 1158 raft_consensus.cc:357] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.596258 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e8d65c2ed5240fc923a7c7b59488e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.597548 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1eaf0266c04873b185c1c0c2da201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.598017 1030 tablet_service.cc:1468] Processing CreateTablet for tablet aec412a15b964060a24631e7ebcf1ef7 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.598407 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec412a15b964060a24631e7ebcf1ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.598906 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1770f6d5d1d4899a66ec23ab4eadc35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.599496 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec412a15b964060a24631e7ebcf1ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.599299 1158 raft_consensus.cc:383] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.600180 1166 ts_tablet_manager.cc:1918] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.600368 1166 log.cc:1199] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e162d6b0fc9e42f8b65446dc6efeb0b5
I20250625 07:52:12.600855 1166 ts_tablet_manager.cc:1939] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.601434 1160 ts_tablet_manager.cc:1905] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.604233 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c (DEFAULT_TABLE table=default.Table_ac4400688af8429ea02aeec0e8d6b03b [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.608654 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (table default.Table_cea9385219bd4aadaa7ea96f51c8bfee [id=65a6d16ff5e444e286e0ca97322594c8]) successfully deleted
I20250625 07:52:12.611121 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d51a898202154881ab4ad41da1d519a3
I20250625 07:52:12.611865 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9e800d4982fa45f4b931985c8b52719d
I20250625 07:52:12.612572 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9916612cb0234ec3a6df8f22c2b50e7c
I20250625 07:52:12.614264 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:12.614527 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 94ef72e8a4f54fd4994dcf4725337478 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94ef72e8a4f54fd4994dcf4725337478 already in progress: creating tablet
I20250625 07:52:12.614917 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.615249 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.615360 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5454db9f8f894091981c0f6abdfd2955" } modify_external_catalogs: true
I20250625 07:52:12.615546 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5454db9f8f894091981c0f6abdfd2955" } modify_external_catalogs: true
W20250625 07:52:12.619154 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a558e0a5403d4489892dfce9587e090c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a558e0a5403d4489892dfce9587e090c already in progress: creating tablet
I20250625 07:52:12.619479 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.619632 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.619693 743 tablet_service.cc:1468] Processing CreateTablet for tablet aec412a15b964060a24631e7ebcf1ef7 (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.619920 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b71aadf074a4a39943b2ff930909f93. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.620265 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3338d1c708b4421c8501740da89afec5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3338d1c708b4421c8501740da89afec5 already in progress: creating tablet
I20250625 07:52:12.620172 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 32bd7cdef96946e88683e457c2f26c9b (DEFAULT_TABLE table=default.Table_4efdc728e9ca49549585528cc7eb22ee [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.620486 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b1eaf0266c04873b185c1c0c2da201c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.623318 1162 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.623489 1162 ts_tablet_manager.cc:1397] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.004s
I20250625 07:52:12.623828 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.624070 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.624253 1162 raft_consensus.cc:357] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.624387 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.624441 1162 raft_consensus.cc:383] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.624553 1162 raft_consensus.cc:738] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.624080 1158 raft_consensus.cc:738] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.624786 1162 consensus_queue.cc:260] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.625267 1162 ts_tablet_manager.cc:1428] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:12.625561 1162 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.627245 1162 tablet_bootstrap.cc:654] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.628861 1161 ts_tablet_manager.cc:1918] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.628154 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.630303 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.633179 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:12.635063 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.635053 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:12.635879 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1222f2c1027b42c6ad9627359a93d678 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1222f2c1027b42c6ad9627359a93d678 already in progress: creating tablet
W20250625 07:52:12.636184 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 572b063d66434e4796f6b86f8221e3c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 572b063d66434e4796f6b86f8221e3c9 already in progress: creating tablet
I20250625 07:52:12.636432 1158 consensus_queue.cc:260] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.637759 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.637683 1161 log.cc:1199] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3471c83e015b44adbe8224c17a1c2eaf
I20250625 07:52:12.638990 1161 ts_tablet_manager.cc:1939] T 3471c83e015b44adbe8224c17a1c2eaf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:12.639799 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2b020ff9f04b4d1796af4ff585cc04aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b020ff9f04b4d1796af4ff585cc04aa already in progress: creating tablet
I20250625 07:52:12.640748 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3471c83e015b44adbe8224c17a1c2eaf (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:12.643052 1158 ts_tablet_manager.cc:1428] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.051s user 0.001s sys 0.001s
W20250625 07:52:12.644088 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2b020ff9f04b4d1796af4ff585cc04aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b020ff9f04b4d1796af4ff585cc04aa already in progress: creating tablet
W20250625 07:52:12.644323 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 55d190e9d11c4589a82fcc7307e9eccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d190e9d11c4589a82fcc7307e9eccb already in progress: creating tablet
W20250625 07:52:12.644616 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 012df7beedb1446ca5e533d6624afc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012df7beedb1446ca5e533d6624afc10 already in progress: creating tablet
I20250625 07:52:12.644935 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec412a15b964060a24631e7ebcf1ef7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.644914 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d51a898202154881ab4ad41da1d519a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51a898202154881ab4ad41da1d519a3 already in progress: creating tablet
W20250625 07:52:12.645318 583 catalog_manager.cc:4895] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9e800d4982fa45f4b931985c8b52719d because the tablet was not found. No further retry: Not found: Tablet not found: 9e800d4982fa45f4b931985c8b52719d
W20250625 07:52:12.644860 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 55d190e9d11c4589a82fcc7307e9eccb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55d190e9d11c4589a82fcc7307e9eccb already in progress: creating tablet
W20250625 07:52:12.645560 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9916612cb0234ec3a6df8f22c2b50e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9916612cb0234ec3a6df8f22c2b50e7c already in progress: creating tablet
W20250625 07:52:12.645802 583 catalog_manager.cc:4895] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8e8d65c2ed5240fc923a7c7b59488e6a because the tablet was not found. No further retry: Not found: Tablet not found: 8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:12.647117 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a95c624abd8a45079c89fe1750faafcf with delete_type TABLET_DATA_DELETED (replaced by tablet b0d31993b4394cfd8832c93ab31b227b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.648813 1158 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.650241 1161 tablet_replica.cc:331] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:12.651556 1161 raft_consensus.cc:2241] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.651885 1160 ts_tablet_manager.cc:1918] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.652355 1160 log.cc:1199] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/bf0189f6651347a18663a94aa61ee267
I20250625 07:52:12.652457 1161 raft_consensus.cc:2270] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.653012 1160 ts_tablet_manager.cc:1939] T bf0189f6651347a18663a94aa61ee267 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.654328 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet bf0189f6651347a18663a94aa61ee267 (table default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]) successfully deleted
I20250625 07:52:12.654608 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 95a763abda074d358f176fd7e7be0e5c (DEFAULT_TABLE table=default.Table_1b354149a4d24041b4a1a28faa3782d7 [id=44ba0323288e45269c5dda7cd7f60f84]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:12.654834 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9e800d4982fa45f4b931985c8b52719d because the tablet was not found. No further retry: Not found: Tablet not found: 9e800d4982fa45f4b931985c8b52719d
I20250625 07:52:12.604343 889 tablet_service.cc:1468] Processing CreateTablet for tablet 95a763abda074d358f176fd7e7be0e5c (DEFAULT_TABLE table=default.Table_1b354149a4d24041b4a1a28faa3782d7 [id=44ba0323288e45269c5dda7cd7f60f84]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:12.655062 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9916612cb0234ec3a6df8f22c2b50e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9916612cb0234ec3a6df8f22c2b50e7c already in progress: creating tablet
W20250625 07:52:12.655238 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d51a898202154881ab4ad41da1d519a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51a898202154881ab4ad41da1d519a3 already in progress: creating tablet
I20250625 07:52:12.655001 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32bd7cdef96946e88683e457c2f26c9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.655465 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8e8d65c2ed5240fc923a7c7b59488e6a because the tablet was not found. No further retry: Not found: Tablet not found: 8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:12.655942 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81396ee6ed7400baa30dcc5d71ba45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.656124 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a763abda074d358f176fd7e7be0e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.658200 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a763abda074d358f176fd7e7be0e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.658277 1158 tablet_bootstrap.cc:654] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.662768 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de1492e483e47919ee5581ecee227be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.662997 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 9916612cb0234ec3a6df8f22c2b50e7c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.669001 1161 ts_tablet_manager.cc:1905] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.675300 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 6fc7c0feb0e04ef28c323323e3007efc from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.680413 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e800d4982fa45f4b931985c8b52719d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.680670 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 9916612cb0234ec3a6df8f22c2b50e7c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.686430 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:12.686969 588 catalog_manager.cc:6652] ReplaceTablet: tablet 6fc7c0feb0e04ef28c323323e3007efc deleted and replaced by tablet b769ac57346a427abd22214eebc0d324
I20250625 07:52:12.689750 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eea1c257b584a029efa08922547b0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.690191 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cab9864e01764dbdba79268deb43905c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.690438 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0125dcbb680481899e4adad86d7c2b5" } modify_external_catalogs: true
I20250625 07:52:12.690623 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0125dcbb680481899e4adad86d7c2b5" } modify_external_catalogs: true
W20250625 07:52:12.690809 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cab9864e01764dbdba79268deb43905c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.691164 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6de1678c755f45adb2f7bd456730a44e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.691684 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6de1678c755f45adb2f7bd456730a44e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.691972 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_36239d41a07544179db0cf4f2fddf2f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.692417 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36239d41a07544179db0cf4f2fddf2f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.692907 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_658480d4343a409bad94a3ecbecb484b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.693311 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_658480d4343a409bad94a3ecbecb484b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.693487 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1a4d7facdef948f894c8b729cdfb82f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.694401 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a4d7facdef948f894c8b729cdfb82f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.695077 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e01dd6d63c224ada88cf0958a6f62f1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.695530 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e01dd6d63c224ada88cf0958a6f62f1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.695473 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9c43e2a9a4474aed9941962b1623bcaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.697067 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c43e2a9a4474aed9941962b1623bcaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.699208 1163 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.700102 1163 ts_tablet_manager.cc:1397] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.158s user 0.003s sys 0.024s
I20250625 07:52:12.700244 1161 ts_tablet_manager.cc:1918] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.700382 1161 log.cc:1199] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a95c624abd8a45079c89fe1750faafcf
I20250625 07:52:12.700815 1161 ts_tablet_manager.cc:1939] T a95c624abd8a45079c89fe1750faafcf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:12.701125 1163 raft_consensus.cc:357] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.701331 1163 raft_consensus.cc:383] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.701418 1163 raft_consensus.cc:738] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.701619 1163 consensus_queue.cc:260] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.702054 1163 ts_tablet_manager.cc:1428] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:12.702100 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.702364 1163 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:12.702661 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.707000 1163 tablet_bootstrap.cc:654] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.707405 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a95c624abd8a45079c89fe1750faafcf (table default.Table_a2747cfb800e440eb843d823139c4f27 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:12.708065 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.708503 893 tablet_service.cc:1468] Processing CreateTablet for tablet b769ac57346a427abd22214eebc0d324 (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.708617 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b769ac57346a427abd22214eebc0d324 (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.708902 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b769ac57346a427abd22214eebc0d324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.709019 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b769ac57346a427abd22214eebc0d324. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.715222 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6fc7c0feb0e04ef28c323323e3007efc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fc7c0feb0e04ef28c323323e3007efc already in progress: creating tablet
I20250625 07:52:12.715816 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.716084 1160 tablet_replica.cc:331] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.716240 1160 raft_consensus.cc:2241] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.716424 1160 raft_consensus.cc:2270] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.717240 1160 ts_tablet_manager.cc:1905] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.719049 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 239e99c6b91d41d68a213a4c8d79470f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.655328 1166 tablet_replica.cc:331] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.685909 1162 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.721299 1162 ts_tablet_manager.cc:1397] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.008s
I20250625 07:52:12.721510 752 tablet_service.cc:1468] Processing CreateTablet for tablet 95a763abda074d358f176fd7e7be0e5c (DEFAULT_TABLE table=default.Table_1b354149a4d24041b4a1a28faa3782d7 [id=44ba0323288e45269c5dda7cd7f60f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.721946 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a763abda074d358f176fd7e7be0e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.722157 1162 raft_consensus.cc:357] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.722232 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d138706726e44916b4369a0f255a420c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.722378 1162 raft_consensus.cc:383] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.722452 1162 raft_consensus.cc:738] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.722703 1162 consensus_queue.cc:260] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:12.722806 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d138706726e44916b4369a0f255a420c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.689064 1171 raft_consensus.cc:491] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:12.723871 1171 raft_consensus.cc:513] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.724772 1166 raft_consensus.cc:2241] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.725057 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.726109 1171 leader_election.cc:290] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:12.706571 749 tablet_service.cc:1468] Processing CreateTablet for tablet b769ac57346a427abd22214eebc0d324 (DEFAULT_TABLE table=default.Table_22595703a6814afeaeb032090df4c1a3 [id=1dd98ab8da6243468e3439bdeea865eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.727197 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b769ac57346a427abd22214eebc0d324. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.729565 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b354149a4d24041b4a1a28faa3782d7" } new_table_name: "default.Table_3f4a64baf13c4985b561eae647c888bd" modify_external_catalogs: true
I20250625 07:52:12.731503 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a2747cfb800e440eb843d823139c4f27" } new_table_name: "default.Table_305bb3fb930b43aabcff2b977e0ff866" modify_external_catalogs: true
I20250625 07:52:12.726106 1166 raft_consensus.cc:2270] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.732789 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6de1678c755f45adb2f7bd456730a44e" } new_table_name: "default.Table_2131dbf2d90d4c45807fef063380c5ee" modify_external_catalogs: true
W20250625 07:52:12.735016 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6fc7c0feb0e04ef28c323323e3007efc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fc7c0feb0e04ef28c323323e3007efc already in progress: creating tablet
I20250625 07:52:12.737741 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:12.714233 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.740489 1166 ts_tablet_manager.cc:1905] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.740931 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cab9864e01764dbdba79268deb43905c" } new_table_name: "default.Table_2064fe78f7a74f389240146e3502eb6c" modify_external_catalogs: true
I20250625 07:52:12.742591 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_22595703a6814afeaeb032090df4c1a3" } new_table_name: "default.Table_08a3260b659949be80df2e4a6bd81cad" modify_external_catalogs: true
I20250625 07:52:12.744410 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_53870f55fcc5414ab6e8ccb92db24344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.745126 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53870f55fcc5414ab6e8ccb92db24344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.750496 1160 ts_tablet_manager.cc:1918] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.751847 1160 log.cc:1199] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9916612cb0234ec3a6df8f22c2b50e7c
I20250625 07:52:12.752381 1160 ts_tablet_manager.cc:1939] T 9916612cb0234ec3a6df8f22c2b50e7c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.751709 1163 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.753775 1163 ts_tablet_manager.cc:1397] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.003s
I20250625 07:52:12.754880 1163 raft_consensus.cc:357] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.755596 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 3338d1c708b4421c8501740da89afec5 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:12.756090 1163 raft_consensus.cc:383] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.756182 1163 raft_consensus.cc:738] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.756302 1162 ts_tablet_manager.cc:1428] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.035s user 0.004s sys 0.006s
I20250625 07:52:12.756446 1163 consensus_queue.cc:260] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.757542 1163 ts_tablet_manager.cc:1428] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:12.757846 1163 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.758289 1162 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.760052 1162 tablet_bootstrap.cc:654] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.762671 1158 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.762893 1158 ts_tablet_manager.cc:1397] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.001s
I20250625 07:52:12.763844 1158 raft_consensus.cc:357] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.764068 1158 raft_consensus.cc:383] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.764163 1158 raft_consensus.cc:738] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.764443 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.764848 1158 consensus_queue.cc:260] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.765419 1158 ts_tablet_manager.cc:1428] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:12.765725 1158 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.767513 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.767601 1158 tablet_bootstrap.cc:654] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.768936 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9916612cb0234ec3a6df8f22c2b50e7c (table default.Table_9fdc22d5be4b43a4be699388015b9864 [id=7fbebf8268534597bd15d2bc70204cb7]) successfully deleted
I20250625 07:52:12.772852 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.773559 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.773860 1163 tablet_bootstrap.cc:654] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.775367 1160 tablet_replica.cc:331] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.775679 1160 raft_consensus.cc:2241] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.775947 1160 raft_consensus.cc:2270] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:12.776341 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 239e99c6b91d41d68a213a4c8d79470f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239e99c6b91d41d68a213a4c8d79470f already in progress: creating tablet
I20250625 07:52:12.777786 1160 ts_tablet_manager.cc:1905] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.783380 1166 ts_tablet_manager.cc:1918] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.783557 1166 log.cc:1199] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:12.783929 1166 ts_tablet_manager.cc:1939] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.784757 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3e14b60ee7a843e5bb878148d8fd2e53 (table default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]) successfully deleted
W20250625 07:52:12.785192 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 012df7beedb1446ca5e533d6624afc10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012df7beedb1446ca5e533d6624afc10 already in progress: creating tablet
I20250625 07:52:12.791640 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:12.792418 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 239e99c6b91d41d68a213a4c8d79470f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239e99c6b91d41d68a213a4c8d79470f already in progress: creating tablet
W20250625 07:52:12.797559 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95a763abda074d358f176fd7e7be0e5c consensus state has no leader
W20250625 07:52:12.798130 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d31993b4394cfd8832c93ab31b227b consensus state has no leader
W20250625 07:52:12.798552 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c8c317c10e14332a3b3919808e4a512 has no consensus state
W20250625 07:52:12.798656 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9916612cb0234ec3a6df8f22c2b50e7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9916612cb0234ec3a6df8f22c2b50e7c already in progress: creating tablet
I20250625 07:52:12.798928 1166 tablet_replica.cc:331] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:12.798951 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a53d8f50224460b932f4f44659336da has no consensus state
I20250625 07:52:12.799139 1166 raft_consensus.cc:2241] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:12.799130 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9e800d4982fa45f4b931985c8b52719d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e800d4982fa45f4b931985c8b52719d already in progress: creating tablet
I20250625 07:52:12.799340 1166 raft_consensus.cc:2270] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:12.799342 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b769ac57346a427abd22214eebc0d324 consensus state has no leader
W20250625 07:52:12.799373 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d51a898202154881ab4ad41da1d519a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d51a898202154881ab4ad41da1d519a3 already in progress: creating tablet
I20250625 07:52:12.800628 1158 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.800819 1158 ts_tablet_manager.cc:1397] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20250625 07:52:12.801621 1158 raft_consensus.cc:357] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.801826 1158 raft_consensus.cc:383] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.801909 1158 raft_consensus.cc:738] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.802191 1158 consensus_queue.cc:260] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.802695 1158 ts_tablet_manager.cc:1428] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:12.802960 1158 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.803543 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0452e73865964f709310fd22ac39a5f0 (DEFAULT_TABLE table=default.Table_36239d41a07544179db0cf4f2fddf2f8 [id=7ce5c0adc972450bb89be40d18e4cc30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.803948 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0452e73865964f709310fd22ac39a5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.804222 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7a53d8f50224460b932f4f44659336da (DEFAULT_TABLE table=default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.804643 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a53d8f50224460b932f4f44659336da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.804657 1158 tablet_bootstrap.cc:654] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.805172 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet e90434606299495081d752df4761f9f8 (table default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]). Expected version 1 got 0
I20250625 07:52:12.806353 1166 ts_tablet_manager.cc:1905] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.807426 1163 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.808254 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3c8c317c10e14332a3b3919808e4a512 (DEFAULT_TABLE table=default.Table_2131dbf2d90d4c45807fef063380c5ee [id=152fbc75a19b4eacb4909704af909443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.808590 1163 ts_tablet_manager.cc:1397] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.003s
I20250625 07:52:12.808738 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c8c317c10e14332a3b3919808e4a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.809571 1163 raft_consensus.cc:357] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.812951 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e87722f6f3e42048774f5c2a33ff506" } modify_external_catalogs: true
I20250625 07:52:12.813194 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e87722f6f3e42048774f5c2a33ff506" } modify_external_catalogs: true
I20250625 07:52:12.810721 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0452e73865964f709310fd22ac39a5f0 (DEFAULT_TABLE table=default.Table_36239d41a07544179db0cf4f2fddf2f8 [id=7ce5c0adc972450bb89be40d18e4cc30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.813973 1163 raft_consensus.cc:383] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.814091 1163 raft_consensus.cc:738] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.814401 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2064fe78f7a74f389240146e3502eb6c" } modify_external_catalogs: true
I20250625 07:52:12.814349 1163 consensus_queue.cc:260] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.814571 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2064fe78f7a74f389240146e3502eb6c" } modify_external_catalogs: true
I20250625 07:52:12.814633 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_08a3260b659949be80df2e4a6bd81cad" } modify_external_catalogs: true
I20250625 07:52:12.811200 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3c8c317c10e14332a3b3919808e4a512 (DEFAULT_TABLE table=default.Table_2131dbf2d90d4c45807fef063380c5ee [id=152fbc75a19b4eacb4909704af909443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.814810 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_08a3260b659949be80df2e4a6bd81cad" } modify_external_catalogs: true
I20250625 07:52:12.814358 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0452e73865964f709310fd22ac39a5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.817342 1158 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.817482 1163 ts_tablet_manager.cc:1428] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 07:52:12.817512 1158 ts_tablet_manager.cc:1397] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250625 07:52:12.818249 1158 raft_consensus.cc:357] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.818423 1158 raft_consensus.cc:383] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.818480 1158 raft_consensus.cc:738] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.818723 1158 consensus_queue.cc:260] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.819211 1158 ts_tablet_manager.cc:1428] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:12.819274 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c8c317c10e14332a3b3919808e4a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.819502 1158 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.819865 1163 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.821168 1158 tablet_bootstrap.cc:654] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.822160 1163 tablet_bootstrap.cc:654] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.827744 1166 ts_tablet_manager.cc:1918] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.827895 1166 log.cc:1199] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:12.827842 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7a53d8f50224460b932f4f44659336da (DEFAULT_TABLE table=default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.828228 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a53d8f50224460b932f4f44659336da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.828745 1166 ts_tablet_manager.cc:1939] T 94ef72e8a4f54fd4994dcf4725337478 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.830176 1166 tablet_replica.cc:331] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.830400 1166 raft_consensus.cc:2241] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.830591 1166 raft_consensus.cc:2270] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.830950 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 7a53d8f50224460b932f4f44659336da (DEFAULT_TABLE table=default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.831351 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a53d8f50224460b932f4f44659336da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.831971 1158 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.832106 1158 ts_tablet_manager.cc:1397] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250625 07:52:12.832962 1158 raft_consensus.cc:357] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.833156 1158 raft_consensus.cc:383] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.833415 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 94ef72e8a4f54fd4994dcf4725337478 (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
W20250625 07:52:12.833643 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8e8d65c2ed5240fc923a7c7b59488e6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e8d65c2ed5240fc923a7c7b59488e6a already in progress: creating tablet
W20250625 07:52:12.833817 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6fc7c0feb0e04ef28c323323e3007efc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fc7c0feb0e04ef28c323323e3007efc already in progress: creating tablet
W20250625 07:52:12.833981 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 239e99c6b91d41d68a213a4c8d79470f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 239e99c6b91d41d68a213a4c8d79470f already in progress: creating tablet
I20250625 07:52:12.834993 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_994604bdceea483cb2c00ad1e1c642c5" } modify_external_catalogs: true
I20250625 07:52:12.835186 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_994604bdceea483cb2c00ad1e1c642c5" } modify_external_catalogs: true
I20250625 07:52:12.836170 1166 ts_tablet_manager.cc:1905] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.842770 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 3c8c317c10e14332a3b3919808e4a512 (DEFAULT_TABLE table=default.Table_2131dbf2d90d4c45807fef063380c5ee [id=152fbc75a19b4eacb4909704af909443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.843171 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c8c317c10e14332a3b3919808e4a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.843178 752 tablet_service.cc:1468] Processing CreateTablet for tablet c6031b9695cd4a99bd0e5391663170ff (DEFAULT_TABLE table=default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.843561 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6031b9695cd4a99bd0e5391663170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.844058 1166 ts_tablet_manager.cc:1918] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.844427 1166 log.cc:1199] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:12.844859 1166 ts_tablet_manager.cc:1939] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:12.845727 755 tablet_service.cc:1468] Processing CreateTablet for tablet e76df9405d91488dad0cfa7f8755bb1c (DEFAULT_TABLE table=default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.836756 1163 tablet_bootstrap.cc:492] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.845817 743 tablet_service.cc:1468] Processing CreateTablet for tablet 53c6c51a90754878977aeeaacf4773a6 (DEFAULT_TABLE table=default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.846031 1163 ts_tablet_manager.cc:1397] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250625 07:52:12.846087 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76df9405d91488dad0cfa7f8755bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.846374 749 tablet_service.cc:1468] Processing CreateTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 (DEFAULT_TABLE table=default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.846709 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59df0f66116e4dfd9386d107cab0aaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.846858 1163 raft_consensus.cc:357] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.847043 1163 raft_consensus.cc:383] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.847121 1163 raft_consensus.cc:738] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.847357 1163 consensus_queue.cc:260] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.848834 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c6031b9695cd4a99bd0e5391663170ff (DEFAULT_TABLE table=default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.849372 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6031b9695cd4a99bd0e5391663170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.852577 890 tablet_service.cc:1468] Processing CreateTablet for tablet c6031b9695cd4a99bd0e5391663170ff (DEFAULT_TABLE table=default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.833242 1158 raft_consensus.cc:738] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.853001 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6031b9695cd4a99bd0e5391663170ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.853000 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4914a64e704a4386b8241e95229c3786 (DEFAULT_TABLE table=default.Table_e01dd6d63c224ada88cf0958a6f62f1d [id=1aeded17a4f5457bb3eb4b9fa9f56f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.853403 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4914a64e704a4386b8241e95229c3786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.854522 1163 ts_tablet_manager.cc:1428] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:12.857131 888 tablet_service.cc:1468] Processing CreateTablet for tablet 53c6c51a90754878977aeeaacf4773a6 (DEFAULT_TABLE table=default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.857753 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c6c51a90754878977aeeaacf4773a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.859700 1158 consensus_queue.cc:260] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.860637 1158 ts_tablet_manager.cc:1428] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250625 07:52:12.860715 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c6c51a90754878977aeeaacf4773a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.860971 1158 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.861387 887 tablet_service.cc:1468] Processing CreateTablet for tablet 4914a64e704a4386b8241e95229c3786 (DEFAULT_TABLE table=default.Table_e01dd6d63c224ada88cf0958a6f62f1d [id=1aeded17a4f5457bb3eb4b9fa9f56f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.861495 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4384bc544af149338514d40adf683f34 (DEFAULT_TABLE table=default.Table_1a4d7facdef948f894c8b729cdfb82f6 [id=fd174ee9d81e42c19cab3c058584ea20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.861908 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4384bc544af149338514d40adf683f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.861930 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc (DEFAULT_TABLE table=default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.862345 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c180785b64f7d85863c14cfd09ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.865042 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (table default.Table_48c1aaaa0128418cb49437cb4d940dda [id=15c681d0005742e781fe94d2c8adc9d9]) successfully deleted
I20250625 07:52:12.866988 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1eab049ad7fc4b5581e5c31a14491306
I20250625 07:52:12.867658 1163 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.867938 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc (DEFAULT_TABLE table=default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.869774 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c180785b64f7d85863c14cfd09ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.870599 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.871977 885 tablet_service.cc:1468] Processing CreateTablet for tablet e76df9405d91488dad0cfa7f8755bb1c (DEFAULT_TABLE table=default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:12.875226 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1eab049ad7fc4b5581e5c31a14491306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eab049ad7fc4b5581e5c31a14491306 already in progress: creating tablet
I20250625 07:52:12.877753 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7a53d8f50224460b932f4f44659336da
I20250625 07:52:12.878122 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e90434606299495081d752df4761f9f8
I20250625 07:52:12.878532 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b769ac57346a427abd22214eebc0d324
I20250625 07:52:12.878877 884 tablet_service.cc:1468] Processing CreateTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 (DEFAULT_TABLE table=default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.880987 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76df9405d91488dad0cfa7f8755bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.880898 893 tablet_service.cc:1468] Processing CreateTablet for tablet 4384bc544af149338514d40adf683f34 (DEFAULT_TABLE table=default.Table_1a4d7facdef948f894c8b729cdfb82f6 [id=fd174ee9d81e42c19cab3c058584ea20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.881325 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 53c6c51a90754878977aeeaacf4773a6 (DEFAULT_TABLE table=default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.882145 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e90434606299495081d752df4761f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.882855 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.885578 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.887213 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e76df9405d91488dad0cfa7f8755bb1c (DEFAULT_TABLE table=default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.887692 1160 ts_tablet_manager.cc:1918] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:12.888206 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.889045 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59df0f66116e4dfd9386d107cab0aaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.889456 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.893671 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc (DEFAULT_TABLE table=default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.893709 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 4384bc544af149338514d40adf683f34 (DEFAULT_TABLE table=default.Table_1a4d7facdef948f894c8b729cdfb82f6 [id=fd174ee9d81e42c19cab3c058584ea20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.894069 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.894431 1163 tablet_bootstrap.cc:654] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.895161 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.895642 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4914a64e704a4386b8241e95229c3786. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:12.896660 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b769ac57346a427abd22214eebc0d324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b769ac57346a427abd22214eebc0d324 already in progress: creating tablet
I20250625 07:52:12.897035 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c6c51a90754878977aeeaacf4773a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.897145 1158 tablet_bootstrap.cc:654] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.897549 1160 log.cc:1199] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:12.897747 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e162d6b0fc9e42f8b65446dc6efeb0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.898288 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4384bc544af149338514d40adf683f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.901104 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.901476 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e90434606299495081d752df4761f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.901849 1161 tablet_replica.cc:331] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:12.902424 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7a53d8f50224460b932f4f44659336da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a53d8f50224460b932f4f44659336da already in progress: creating tablet
I20250625 07:52:12.902772 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4384bc544af149338514d40adf683f34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.903724 1161 raft_consensus.cc:2241] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.904003 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0452e73865964f709310fd22ac39a5f0 (DEFAULT_TABLE table=default.Table_36239d41a07544179db0cf4f2fddf2f8 [id=7ce5c0adc972450bb89be40d18e4cc30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.904562 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e90434606299495081d752df4761f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:12.905249 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e90434606299495081d752df4761f9f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e90434606299495081d752df4761f9f8 already in progress: creating tablet
W20250625 07:52:12.905490 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b769ac57346a427abd22214eebc0d324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b769ac57346a427abd22214eebc0d324 already in progress: creating tablet
W20250625 07:52:12.906562 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7a53d8f50224460b932f4f44659336da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a53d8f50224460b932f4f44659336da already in progress: creating tablet
I20250625 07:52:12.906821 1162 tablet_bootstrap.cc:492] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.902501 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 (DEFAULT_TABLE table=default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.907020 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.905270 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.908000 1162 ts_tablet_manager.cc:1397] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.150s user 0.027s sys 0.018s
I20250625 07:52:12.908851 1162 raft_consensus.cc:357] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.909210 1162 raft_consensus.cc:383] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.909348 1162 raft_consensus.cc:738] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.909643 1162 consensus_queue.cc:260] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.910192 1162 ts_tablet_manager.cc:1428] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:12.910867 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3c180785b64f7d85863c14cfd09ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.910832 1161 raft_consensus.cc:2270] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.911331 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:12.912451 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.913892 1162 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:12.913908 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e14b60ee7a843e5bb878148d8fd2e53" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:12.914994 713 leader_election.cc:343] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Illegal state: Tablet not RUNNING: STOPPED
I20250625 07:52:12.915701 1162 tablet_bootstrap.cc:654] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:12.916822 752 tablet_service.cc:1515] Processing DeleteTablet for tablet bf0189f6651347a18663a94aa61ee267 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:12.917390 1166 tablet_replica.cc:331] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:12.917547 1166 raft_consensus.cc:2241] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.917730 1166 raft_consensus.cc:2270] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.921706 1166 ts_tablet_manager.cc:1905] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.924592 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_79f17dafcee94151a9c0ee944c8037f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.925264 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79f17dafcee94151a9c0ee944c8037f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.925434 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ac363395568d469aa75b630fad70e237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.925629 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.926153 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:12.926195 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac363395568d469aa75b630fad70e237 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.926685 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bcddc07c3a8a46449f20441de84992f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.927091 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcddc07c3a8a46449f20441de84992f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.932642 1160 ts_tablet_manager.cc:1939] T 3e14b60ee7a843e5bb878148d8fd2e53 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:12.936168 1160 tablet_replica.cc:331] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:12.936364 1160 raft_consensus.cc:2241] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:12.936342 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3e14b60ee7a843e5bb878148d8fd2e53 (table default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]) successfully deleted
I20250625 07:52:12.936836 1160 raft_consensus.cc:2270] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:12.957036 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a4e6179f3fa6451099d6493d6182ba85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.957768 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4e6179f3fa6451099d6493d6182ba85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.958494 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f54f59a06d1e4cad83c44dd16452812d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.959223 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f54f59a06d1e4cad83c44dd16452812d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.960585 1160 ts_tablet_manager.cc:1905] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.960834 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b9d2937a36aa4a7daf58b3c859ab9c94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:12.961700 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c43e2a9a4474aed9941962b1623bcaa" } modify_external_catalogs: true
I20250625 07:52:12.962024 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1635d57f0e2c477daa163a4fcedfae91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.962524 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.963717 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c43e2a9a4474aed9941962b1623bcaa" } modify_external_catalogs: true
W20250625 07:52:12.964191 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1635d57f0e2c477daa163a4fcedfae91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.969841 1163 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:12.970618 1163 ts_tablet_manager.cc:1397] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.016s
I20250625 07:52:12.976224 1163 raft_consensus.cc:357] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.976894 1163 raft_consensus.cc:383] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.978637 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_014f0dfc938843338f50edc64d69b63a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:12.979250 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_014f0dfc938843338f50edc64d69b63a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:12.979262 1163 raft_consensus.cc:738] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.980080 1163 consensus_queue.cc:260] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.980867 1158 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:12.981009 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59df0f66116e4dfd9386d107cab0aaf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.981680 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e76df9405d91488dad0cfa7f8755bb1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.983451 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4914a64e704a4386b8241e95229c3786 (DEFAULT_TABLE table=default.Table_e01dd6d63c224ada88cf0958a6f62f1d [id=1aeded17a4f5457bb3eb4b9fa9f56f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:12.983693 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:12.984299 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0452e73865964f709310fd22ac39a5f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.985033 1161 ts_tablet_manager.cc:1905] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:12.985199 1158 ts_tablet_manager.cc:1397] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.000s
I20250625 07:52:12.986132 1158 raft_consensus.cc:357] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.986269 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4914a64e704a4386b8241e95229c3786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:12.986346 1158 raft_consensus.cc:383] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.986446 1158 raft_consensus.cc:738] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.986694 1158 consensus_queue.cc:260] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.987243 1158 ts_tablet_manager.cc:1428] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:12.987578 1158 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:12.982846 1163 ts_tablet_manager.cc:1428] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20250625 07:52:12.991277 1163 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:12.997362 1162 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:12.997601 1162 ts_tablet_manager.cc:1397] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.084s user 0.023s sys 0.025s
I20250625 07:52:12.998975 1162 raft_consensus.cc:357] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:12.999322 1162 raft_consensus.cc:383] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:12.999430 1162 raft_consensus.cc:738] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:12.999725 1162 consensus_queue.cc:260] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.000977 1158 tablet_bootstrap.cc:654] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.001829 1162 ts_tablet_manager.cc:1428] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:13.002218 1162 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.002473 752 tablet_service.cc:1468] Processing CreateTablet for tablet 76fe692067954856908f936ced332c18 (DEFAULT_TABLE table=default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92 [id=b02af454e5f6454885578ace6c10ad97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.003014 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76fe692067954856908f936ced332c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.004160 1162 tablet_bootstrap.cc:654] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.006016 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ea8e512cd60044b791fb0d82facb1802" } new_table_name: "default.Table_ca7c2bc86d9f4ad1896150b322f2d1b5" modify_external_catalogs: true
I20250625 07:52:13.006850 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_299334c718f547068b2ad26626e7f471" } new_table_name: "default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0" modify_external_catalogs: true
I20250625 07:52:13.008118 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ac3fbcd8a67040aa82930017de78ea14" } new_table_name: "default.Table_9babc704c99d4319bf7f3906ef371e72" modify_external_catalogs: true
I20250625 07:52:13.008883 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_36239d41a07544179db0cf4f2fddf2f8" } new_table_name: "default.Table_a478f6f1acd74278941f0b0ed78656a0" modify_external_catalogs: true
I20250625 07:52:13.010244 757 tablet_service.cc:1468] Processing CreateTablet for tablet d92962b9e7d14c8d88d775b98e75de35 (DEFAULT_TABLE table=default.Table_79f17dafcee94151a9c0ee944c8037f9 [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.010531 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.011205 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3497bef6f20d4203ab63ce8841a34b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.011483 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0299e3ee1ac47be8d61de35bf8d8fee" } new_table_name: "default.Table_e407dc6ea5d247b1a9c26729116da0e2" modify_external_catalogs: true
I20250625 07:52:13.012387 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.014284 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92962b9e7d14c8d88d775b98e75de35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.016911 1163 tablet_bootstrap.cc:654] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.020678 887 tablet_service.cc:1468] Processing CreateTablet for tablet 76fe692067954856908f936ced332c18 (DEFAULT_TABLE table=default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92 [id=b02af454e5f6454885578ace6c10ad97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.020697 758 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb2aa6dac1f4443be9782aac8f855af (DEFAULT_TABLE table=default.Table_ac363395568d469aa75b630fad70e237 [id=63494920b2074aba9eb07df70fe2aaa7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:13.021420 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3497bef6f20d4203ab63ce8841a34b73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.022555 1166 ts_tablet_manager.cc:1918] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.023187 1166 log.cc:1199] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/bf0189f6651347a18663a94aa61ee267
I20250625 07:52:13.023737 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76fe692067954856908f936ced332c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.023815 1166 ts_tablet_manager.cc:1939] T bf0189f6651347a18663a94aa61ee267 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.024099 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb2aa6dac1f4443be9782aac8f855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.032750 885 tablet_service.cc:1468] Processing CreateTablet for tablet d92962b9e7d14c8d88d775b98e75de35 (DEFAULT_TABLE table=default.Table_79f17dafcee94151a9c0ee944c8037f9 [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.032562 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb2aa6dac1f4443be9782aac8f855af (DEFAULT_TABLE table=default.Table_ac363395568d469aa75b630fad70e237 [id=63494920b2074aba9eb07df70fe2aaa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.033752 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92962b9e7d14c8d88d775b98e75de35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.037664 757 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.042865 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb2aa6dac1f4443be9782aac8f855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.043774 1162 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.043759 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 94ef72e8a4f54fd4994dcf4725337478 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.044001 1162 ts_tablet_manager.cc:1397] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.003s
I20250625 07:52:13.044623 1160 ts_tablet_manager.cc:1918] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.044799 1160 log.cc:1199] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/37f7c060fb8648b9aaa766bbaa1b4ba7
I20250625 07:52:13.045228 1160 ts_tablet_manager.cc:1939] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.046387 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (table default.Table_b7677386183a479f9714604d7c2ab058 [id=e4b3ee29eafd4621811d8db11bdd65c4]) successfully deleted
W20250625 07:52:13.048235 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1eab049ad7fc4b5581e5c31a14491306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eab049ad7fc4b5581e5c31a14491306 already in progress: creating tablet
W20250625 07:52:13.048990 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7a53d8f50224460b932f4f44659336da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a53d8f50224460b932f4f44659336da already in progress: creating tablet
I20250625 07:52:13.049428 1160 tablet_replica.cc:331] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.049697 1160 raft_consensus.cc:2241] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.051909 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 834d516547b046288e58b7cde4c56e5a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.052289 1160 raft_consensus.cc:2270] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.053560 596 master_service.cc:948] ReplaceTablet: received request to replace tablet b769ac57346a427abd22214eebc0d324 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.055778 1160 ts_tablet_manager.cc:1905] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.056039 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 76fe692067954856908f936ced332c18 (DEFAULT_TABLE table=default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92 [id=b02af454e5f6454885578ace6c10ad97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.056105 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d92962b9e7d14c8d88d775b98e75de35 (DEFAULT_TABLE table=default.Table_79f17dafcee94151a9c0ee944c8037f9 [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.057346 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76fe692067954856908f936ced332c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.058382 1163 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.059247 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d92962b9e7d14c8d88d775b98e75de35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.058543 1163 ts_tablet_manager.cc:1397] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20250625 07:52:13.061071 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.063808 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9cb2aa6dac1f4443be9782aac8f855af (DEFAULT_TABLE table=default.Table_ac363395568d469aa75b630fad70e237 [id=63494920b2074aba9eb07df70fe2aaa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.064419 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb2aa6dac1f4443be9782aac8f855af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.063639 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e14b60ee7a843e5bb878148d8fd2e53" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:13.066994 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7139335c7124ab79bc2d0a251984199 consensus state has no leader
W20250625 07:52:13.067790 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0452e73865964f709310fd22ac39a5f0 consensus state has no leader
W20250625 07:52:13.068189 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b3488a9fb19452dac3241170dd2fee2 consensus state has no leader
W20250625 07:52:13.068722 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fdb6a689ce6440ab7037484e2b9cdc5 consensus state has no leader
I20250625 07:52:13.069478 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet bf0189f6651347a18663a94aa61ee267 (table default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]) successfully deleted
I20250625 07:52:13.070087 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d138706726e44916b4369a0f255a420c" } modify_external_catalogs: true
I20250625 07:52:13.070290 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d138706726e44916b4369a0f255a420c" } modify_external_catalogs: true
I20250625 07:52:13.070551 1162 raft_consensus.cc:357] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.070737 1162 raft_consensus.cc:383] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.071099 1160 ts_tablet_manager.cc:1918] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.070840 1162 raft_consensus.cc:738] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:13.071238 713 leader_election.cc:343] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:13.071748 713 leader_election.cc:304] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:13.072067 713 raft_consensus.cc:2684] T 3e14b60ee7a843e5bb878148d8fd2e53 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:13.073110 1162 consensus_queue.cc:260] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.073351 1166 tablet_replica.cc:331] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.073576 1166 raft_consensus.cc:2241] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.073742 1166 raft_consensus.cc:2270] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.076556 1160 log.cc:1199] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e162d6b0fc9e42f8b65446dc6efeb0b5
I20250625 07:52:13.076779 1166 ts_tablet_manager.cc:1905] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.077273 1160 ts_tablet_manager.cc:1939] T e162d6b0fc9e42f8b65446dc6efeb0b5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.077661 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.078202 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 9e800d4982fa45f4b931985c8b52719d from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.078868 1161 ts_tablet_manager.cc:1918] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.079123 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e162d6b0fc9e42f8b65446dc6efeb0b5 (table default.Table_cea9385219bd4aadaa7ea96f51c8bfee [id=65a6d16ff5e444e286e0ca97322594c8]) successfully deleted
I20250625 07:52:13.081214 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0" } modify_external_catalogs: true
I20250625 07:52:13.081341 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b9d2937a36aa4a7daf58b3c859ab9c94" } modify_external_catalogs: true
I20250625 07:52:13.081421 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0" } modify_external_catalogs: true
I20250625 07:52:13.081535 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b9d2937a36aa4a7daf58b3c859ab9c94" } modify_external_catalogs: true
I20250625 07:52:13.082008 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c931105d39a449fdbdc73dc16830a6a4" } modify_external_catalogs: true
I20250625 07:52:13.082180 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c931105d39a449fdbdc73dc16830a6a4" } modify_external_catalogs: true
I20250625 07:52:13.086153 1163 raft_consensus.cc:357] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.086367 1163 raft_consensus.cc:383] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.086445 1163 raft_consensus.cc:738] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.086684 1163 consensus_queue.cc:260] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.087184 1163 ts_tablet_manager.cc:1428] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20250625 07:52:13.087530 1163 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.087960 1158 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:13.088089 1158 ts_tablet_manager.cc:1397] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.006s
I20250625 07:52:13.088212 1161 log.cc:1199] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e90434606299495081d752df4761f9f8
I20250625 07:52:13.089313 1158 raft_consensus.cc:357] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.089481 1158 raft_consensus.cc:383] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.089805 1158 raft_consensus.cc:738] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.090023 1158 consensus_queue.cc:260] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:13.091327 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e90434606299495081d752df4761f9f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e90434606299495081d752df4761f9f8 already in progress: creating tablet
I20250625 07:52:13.091830 1160 tablet_replica.cc:331] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.091989 1160 raft_consensus.cc:2241] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:13.091984 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b769ac57346a427abd22214eebc0d324 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b769ac57346a427abd22214eebc0d324 already in progress: creating tablet
I20250625 07:52:13.092180 1160 raft_consensus.cc:2270] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.093478 1160 ts_tablet_manager.cc:1905] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.095875 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.096159 1163 tablet_bootstrap.cc:654] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.099011 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3eea1c257b584a029efa08922547b0cc consensus state has no leader
I20250625 07:52:13.099167 1161 ts_tablet_manager.cc:1939] T e90434606299495081d752df4761f9f8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:13.100139 1161 tablet_replica.cc:331] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:13.100816 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9c770b41e0844786ad81c7b3579f9934 (DEFAULT_TABLE table=default.Table_a4e6179f3fa6451099d6493d6182ba85 [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.101213 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c770b41e0844786ad81c7b3579f9934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.102638 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa30ca0035048d3b60669994d402734 (DEFAULT_TABLE table=default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.103809 1166 ts_tablet_manager.cc:1918] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.103926 1158 ts_tablet_manager.cc:1428] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 07:52:13.104228 1161 raft_consensus.cc:2241] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.104445 1161 raft_consensus.cc:2270] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.105968 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e90434606299495081d752df4761f9f8 (table default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]) successfully deleted
I20250625 07:52:13.106194 885 tablet_service.cc:1468] Processing CreateTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f (DEFAULT_TABLE table=default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.106700 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f62b03db94d4ee3935866b5dd8bd53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.107631 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (DEFAULT_TABLE table=default.Table_1635d57f0e2c477daa163a4fcedfae91 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.107980 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.109032 893 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa30ca0035048d3b60669994d402734 (DEFAULT_TABLE table=default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.109522 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa30ca0035048d3b60669994d402734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.110215 1167 raft_consensus.cc:491] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.110492 1167 raft_consensus.cc:513] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.115494 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa30ca0035048d3b60669994d402734 (DEFAULT_TABLE table=default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.115875 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa30ca0035048d3b60669994d402734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.119573 1180 raft_consensus.cc:491] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.119833 1180 raft_consensus.cc:513] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.121354 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f (DEFAULT_TABLE table=default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.124229 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e76df9405d91488dad0cfa7f8755bb1c
I20250625 07:52:13.125896 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c6031b9695cd4a99bd0e5391663170ff
I20250625 07:52:13.125896 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.127470 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:13.128336 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.122020 1180 leader_election.cc:290] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.129667 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1e6c2988baa14cbc8c878f79e942880d (DEFAULT_TABLE table=default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.121805 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f62b03db94d4ee3935866b5dd8bd53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.112668 1167 leader_election.cc:290] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:13.113334 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (DEFAULT_TABLE table=default.Table_1635d57f0e2c477daa163a4fcedfae91 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.133045 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.133723 1160 ts_tablet_manager.cc:1918] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.133868 1160 log.cc:1199] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:13.134315 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.134490 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.134343 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.134855 1160 ts_tablet_manager.cc:1939] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.135998 1160 tablet_replica.cc:331] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.136240 1160 raft_consensus.cc:2241] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.136442 1160 raft_consensus.cc:2270] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.136854 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6c2988baa14cbc8c878f79e942880d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.136793 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (table default.Table_48c1aaaa0128418cb49437cb4d940dda [id=15c681d0005742e781fe94d2c8adc9d9]) successfully deleted
I20250625 07:52:13.139492 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b7139335c7124ab79bc2d0a251984199
I20250625 07:52:13.140476 1158 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:13.142275 1158 tablet_bootstrap.cc:654] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.144095 1161 ts_tablet_manager.cc:1905] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.144556 752 tablet_service.cc:1468] Processing CreateTablet for tablet 9c770b41e0844786ad81c7b3579f9934 (DEFAULT_TABLE table=default.Table_a4e6179f3fa6451099d6493d6182ba85 [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.145015 757 tablet_service.cc:1468] Processing CreateTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f (DEFAULT_TABLE table=default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.145079 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0dd2a46a354a435497fe259c42f87353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.145385 758 tablet_service.cc:1468] Processing CreateTablet for tablet 1e6c2988baa14cbc8c878f79e942880d (DEFAULT_TABLE table=default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.145699 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa30ca0035048d3b60669994d402734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.145550 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0f4d48075c0b46f3bfdb0c252203109f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.145764 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dd2a46a354a435497fe259c42f87353 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:13.146231 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f4d48075c0b46f3bfdb0c252203109f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.147364 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7c4e84ad77a74e08a1c7aab848d552b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.147814 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c4e84ad77a74e08a1c7aab848d552b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.148825 1166 log.cc:1199] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/f774b4ac0c774ee7bf33d13a678cd368
I20250625 07:52:13.148785 755 tablet_service.cc:1468] Processing CreateTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (DEFAULT_TABLE table=default.Table_1635d57f0e2c477daa163a4fcedfae91 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.149197 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.149219 1166 ts_tablet_manager.cc:1939] T f774b4ac0c774ee7bf33d13a678cd368 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.152774 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.153119 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6c2988baa14cbc8c878f79e942880d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.153856 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f62b03db94d4ee3935866b5dd8bd53f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.154768 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c770b41e0844786ad81c7b3579f9934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.157011 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.157096 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet f774b4ac0c774ee7bf33d13a678cd368 (table default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]) successfully deleted
W20250625 07:52:13.157447 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9861eea50d944b2c92be67c3c7c34582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9861eea50d944b2c92be67c3c7c34582 already in progress: creating tablet
W20250625 07:52:13.157735 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e76df9405d91488dad0cfa7f8755bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e76df9405d91488dad0cfa7f8755bb1c already in progress: creating tablet
W20250625 07:52:13.157946 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b09dc0b8a496402b82f98daed9af0d12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09dc0b8a496402b82f98daed9af0d12 already in progress: creating tablet
W20250625 07:52:13.158604 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c6031b9695cd4a99bd0e5391663170ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6031b9695cd4a99bd0e5391663170ff already in progress: creating tablet
W20250625 07:52:13.169369 485 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:39098 (request call id 27) took 830 ms (client timeout 1000 ms). Trace:
W20250625 07:52:13.169579 485 rpcz_store.cc:269] 0625 07:52:12.338985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:12.339057 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:13.169263 (+830206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 07:52:13.171999 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.174903 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ceee7626eede43fb91ee9f43f0410e02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.175537 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceee7626eede43fb91ee9f43f0410e02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.128556 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1e6c2988baa14cbc8c878f79e942880d (DEFAULT_TABLE table=default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.177270 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e6c2988baa14cbc8c878f79e942880d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.179831 1160 ts_tablet_manager.cc:1905] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:13.180429 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b7139335c7124ab79bc2d0a251984199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7139335c7124ab79bc2d0a251984199 already in progress: creating tablet
I20250625 07:52:13.185127 1162 ts_tablet_manager.cc:1428] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20250625 07:52:13.187386 1162 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.189028 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b662e261763d437fafa99ee7cade09e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.189720 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b662e261763d437fafa99ee7cade09e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.192243 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5f62b03db94d4ee3935866b5dd8bd53f
I20250625 07:52:13.193313 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:13.194022 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5f62b03db94d4ee3935866b5dd8bd53f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f62b03db94d4ee3935866b5dd8bd53f already in progress: creating tablet
I20250625 07:52:13.194548 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.195669 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_889e54e43d5b47c888f9006ba7ff4f85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.188264 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9c770b41e0844786ad81c7b3579f9934 (DEFAULT_TABLE table=default.Table_a4e6179f3fa6451099d6493d6182ba85 [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:13.196280 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_889e54e43d5b47c888f9006ba7ff4f85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.196406 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c770b41e0844786ad81c7b3579f9934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.196890 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.197173 1166 tablet_replica.cc:331] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.197361 1166 raft_consensus.cc:2241] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.197369 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.197556 1166 raft_consensus.cc:2270] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.197943 1162 tablet_bootstrap.cc:654] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.198185 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.198258 1166 ts_tablet_manager.cc:1905] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.200843 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.201419 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.201980 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.204967 1180 raft_consensus.cc:491] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.205148 1180 raft_consensus.cc:513] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.205734 1180 leader_election.cc:290] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.206491 752 tablet_service.cc:1468] Processing CreateTablet for tablet ebb943df93284913b6a520e099399322 (DEFAULT_TABLE table=default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.207067 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb943df93284913b6a520e099399322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.207888 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_637a8e11bb45485398b914ec9dda3c77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.208417 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_637a8e11bb45485398b914ec9dda3c77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.209968 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_698024fc59a24b3ca555fe517fe384ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.210054 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ebb943df93284913b6a520e099399322 (DEFAULT_TABLE table=default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.210130 1166 ts_tablet_manager.cc:1918] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.210237 1166 log.cc:1199] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9861eea50d944b2c92be67c3c7c34582
I20250625 07:52:13.210500 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb943df93284913b6a520e099399322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.210628 1166 ts_tablet_manager.cc:1939] T 9861eea50d944b2c92be67c3c7c34582 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:13.210713 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_698024fc59a24b3ca555fe517fe384ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.217196 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9861eea50d944b2c92be67c3c7c34582 (table default.Table_dfdde9ef755044f7b3d9ab934ca4b231 [id=cc9a674f99634eae89fb70aad7f6ef99]) successfully deleted
I20250625 07:52:13.217986 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4ea5f6754f394000b657b1deee066ac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.218559 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea5f6754f394000b657b1deee066ac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.222558 1223 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:13.222710 1223 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:13.222946 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu with pid 424
I20250625 07:52:13.225441 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 834d516547b046288e58b7cde4c56e5a
I20250625 07:52:13.225951 597 catalog_manager.cc:6652] ReplaceTablet: tablet 834d516547b046288e58b7cde4c56e5a deleted and replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:13.227607 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 834d516547b046288e58b7cde4c56e5a with delete_type TABLET_DATA_DELETED (replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:13.228298 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 834d516547b046288e58b7cde4c56e5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 834d516547b046288e58b7cde4c56e5a already in progress: creating tablet
I20250625 07:52:13.233738 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 23755f70822740e08ca6a5e5b5571458 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.247428 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 834d516547b046288e58b7cde4c56e5a with delete_type TABLET_DATA_DELETED (replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.249032 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.249117 1158 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:13.249290 1158 ts_tablet_manager.cc:1397] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.005s
I20250625 07:52:13.248961 893 tablet_service.cc:1468] Processing CreateTablet for tablet ebb943df93284913b6a520e099399322 (DEFAULT_TABLE table=default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.250173 1158 raft_consensus.cc:357] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.250411 1158 raft_consensus.cc:383] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.250496 1158 raft_consensus.cc:738] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.251224 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebb943df93284913b6a520e099399322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.254271 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 834d516547b046288e58b7cde4c56e5a with delete_type TABLET_DATA_DELETED (replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.261476 885 tablet_service.cc:1468] Processing CreateTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec (DEFAULT_TABLE table=default.Table_ceee7626eede43fb91ee9f43f0410e02 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.262104 886 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe (DEFAULT_TABLE table=default.Table_0f4d48075c0b46f3bfdb0c252203109f [id=60fd5cf37619424381543d3cdc94167d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.262176 755 tablet_service.cc:1468] Processing CreateTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec (DEFAULT_TABLE table=default.Table_ceee7626eede43fb91ee9f43f0410e02 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.262308 887 tablet_service.cc:1468] Processing CreateTablet for tablet b6382963863548398bed90f03b840806 (DEFAULT_TABLE table=default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.262614 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49697a10cdac4c8ebb69f10f4c3437ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.262876 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (DEFAULT_TABLE table=default.Table_0dd2a46a354a435497fe259c42f87353 [id=e441385dc1044575b17f64b06f14c648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.263320 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49697a10cdac4c8ebb69f10f4c3437ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.265023 757 tablet_service.cc:1468] Processing CreateTablet for tablet b6382963863548398bed90f03b840806 (DEFAULT_TABLE table=default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.265419 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6382963863548398bed90f03b840806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.267360 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3f2915b351407da08a9d6ba8ed402f (DEFAULT_TABLE table=default.Table_3497bef6f20d4203ab63ce8841a34b73 [id=1ab8d615f8a84fdeba96d8266c1d4555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.267637 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f63fdf2c8464c8fb58f412263d4e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.267845 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6382963863548398bed90f03b840806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.268913 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d66546d7c42949dfe5db77f1057fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.269023 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3f2915b351407da08a9d6ba8ed402f (DEFAULT_TABLE table=default.Table_3497bef6f20d4203ab63ce8841a34b73 [id=1ab8d615f8a84fdeba96d8266c1d4555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.269438 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3f2915b351407da08a9d6ba8ed402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.270210 1162 tablet_bootstrap.cc:492] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.270802 1162 ts_tablet_manager.cc:1397] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.003s
I20250625 07:52:13.271394 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_305bb3fb930b43aabcff2b977e0ff866" } new_table_name: "default.Table_98554e0797a84c44bfe6271791b5c975" modify_external_catalogs: true
I20250625 07:52:13.271498 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b6d8ef5e057f49d8b5a15ebb19bd8c92" } new_table_name: "default.Table_527a45a533ca4babab55fa52d117496e" modify_external_catalogs: true
I20250625 07:52:13.272835 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3f2915b351407da08a9d6ba8ed402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.272821 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1a4d7facdef948f894c8b729cdfb82f6" } new_table_name: "default.Table_ddae42a67be34593a367bf6f3d9e8cde" modify_external_catalogs: true
I20250625 07:52:13.272980 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (DEFAULT_TABLE table=default.Table_0dd2a46a354a435497fe259c42f87353 [id=e441385dc1044575b17f64b06f14c648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.273751 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe (DEFAULT_TABLE table=default.Table_0f4d48075c0b46f3bfdb0c252203109f [id=60fd5cf37619424381543d3cdc94167d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.273983 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_698024fc59a24b3ca555fe517fe384ca" } new_table_name: "default.Table_66371f9b27d84da78425fe46ee94cdcf" modify_external_catalogs: true
I20250625 07:52:13.276117 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f63fdf2c8464c8fb58f412263d4e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.276417 1163 tablet_bootstrap.cc:492] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.276588 1163 ts_tablet_manager.cc:1397] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.189s user 0.035s sys 0.023s
I20250625 07:52:13.276718 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d66546d7c42949dfe5db77f1057fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.283878 583 connection.cc:537] client connection to 127.0.104.62:38477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:13.284215 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:52:13.286089 1163 raft_consensus.cc:357] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.286301 1163 raft_consensus.cc:383] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.286413 1163 raft_consensus.cc:738] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.286645 1163 consensus_queue.cc:260] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.287124 1163 ts_tablet_manager.cc:1428] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 07:52:13.288235 1163 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:13.288132 850 connection.cc:537] client connection to 127.0.104.62:38477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:13.291630 961 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:38477 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:52:13.293515 1162 raft_consensus.cc:357] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.295302 1162 raft_consensus.cc:383] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.296221 1163 tablet_bootstrap.cc:654] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.296334 1226 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 07:52:13.292264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38477 (local address 127.0.104.2:38981)
0625 07:52:13.292479 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0625 07:52:13.292500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0625 07:52:13.292615 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0625 07:52:13.295203 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":115}
I20250625 07:52:13.305683 1160 ts_tablet_manager.cc:1918] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.307974 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 23755f70822740e08ca6a5e5b5571458
I20250625 07:52:13.308601 595 catalog_manager.cc:6652] ReplaceTablet: tablet 23755f70822740e08ca6a5e5b5571458 deleted and replaced by tablet 81976db0a2f94128a89842a12f067231
I20250625 07:52:13.309486 1160 log.cc:1199] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/f774b4ac0c774ee7bf33d13a678cd368
I20250625 07:52:13.310571 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:13.311466 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 23755f70822740e08ca6a5e5b5571458 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23755f70822740e08ca6a5e5b5571458 already in progress: creating tablet
W20250625 07:52:13.311441 851 leader_election.cc:343] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: f774b4ac0c774ee7bf33d13a678cd368
I20250625 07:52:13.255710 1158 consensus_queue.cc:260] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.315025 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f774b4ac0c774ee7bf33d13a678cd368" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:13.316248 850 leader_election.cc:343] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:13.312667 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d31993b4394cfd8832c93ab31b227b consensus state has no leader
I20250625 07:52:13.316440 850 leader_election.cc:304] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:13.316850 850 raft_consensus.cc:2684] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:13.312942 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4384bc544af149338514d40adf683f34 consensus state has no leader
W20250625 07:52:13.313156 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76fe692067954856908f936ced332c18 consensus state has no leader
I20250625 07:52:13.297443 1162 raft_consensus.cc:738] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.316286 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a478f6f1acd74278941f0b0ed78656a0" } new_table_name: "default.Table_ab6d78ba7ead4a7f94152de5254f7bf0" modify_external_catalogs: true
I20250625 07:52:13.317658 1160 ts_tablet_manager.cc:1939] T f774b4ac0c774ee7bf33d13a678cd368 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.316490 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b3c9bae2008e48b199a1d0551ec879c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.318910 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3c9bae2008e48b199a1d0551ec879c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:13.320513 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de9ccb0a2d3846228acbc2769786a059 has no consensus state
I20250625 07:52:13.321451 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe (DEFAULT_TABLE table=default.Table_0f4d48075c0b46f3bfdb0c252203109f [id=60fd5cf37619424381543d3cdc94167d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.322261 1158 ts_tablet_manager.cc:1428] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.067s user 0.002s sys 0.001s
I20250625 07:52:13.322260 1162 consensus_queue.cc:260] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.322772 1162 ts_tablet_manager.cc:1428] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.052s user 0.001s sys 0.001s
I20250625 07:52:13.323163 1162 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.324952 1162 tablet_bootstrap.cc:654] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.325603 824 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:38477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111)
I20250625 07:52:13.326784 1163 tablet_bootstrap.cc:492] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.326931 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_17d5751097fa40feaf90d0fb3c97e1b5" } modify_external_catalogs: true
I20250625 07:52:13.327134 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_17d5751097fa40feaf90d0fb3c97e1b5" } modify_external_catalogs: true
I20250625 07:52:13.328052 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_658480d4343a409bad94a3ecbecb484b" } modify_external_catalogs: true
I20250625 07:52:13.328219 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_658480d4343a409bad94a3ecbecb484b" } modify_external_catalogs: true
I20250625 07:52:13.328308 1163 ts_tablet_manager.cc:1397] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.002s
I20250625 07:52:13.328969 1017 tablet_service.cc:1468] Processing CreateTablet for tablet b6382963863548398bed90f03b840806 (DEFAULT_TABLE table=default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.329030 1163 raft_consensus.cc:357] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.329238 1163 raft_consensus.cc:383] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.329232 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec (DEFAULT_TABLE table=default.Table_ceee7626eede43fb91ee9f43f0410e02 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.329314 1163 raft_consensus.cc:738] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.329373 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f54f59a06d1e4cad83c44dd16452812d" } modify_external_catalogs: true
I20250625 07:52:13.329524 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f54f59a06d1e4cad83c44dd16452812d" } modify_external_catalogs: true
I20250625 07:52:13.329556 1163 consensus_queue.cc:260] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.329704 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 9916612cb0234ec3a6df8f22c2b50e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.329924 1166 tablet_replica.cc:331] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.330029 1163 ts_tablet_manager.cc:1428] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:13.330097 1166 raft_consensus.cc:2241] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.330250 1166 raft_consensus.cc:2270] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.330332 1163 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.330881 1166 ts_tablet_manager.cc:1905] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:13.331272 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 47168.3us
I20250625 07:52:13.332865 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet f774b4ac0c774ee7bf33d13a678cd368 (table default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]) successfully deleted
W20250625 07:52:13.333145 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b09dc0b8a496402b82f98daed9af0d12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09dc0b8a496402b82f98daed9af0d12 already in progress: creating tablet
W20250625 07:52:13.333376 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c6031b9695cd4a99bd0e5391663170ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6031b9695cd4a99bd0e5391663170ff already in progress: creating tablet
W20250625 07:52:13.333601 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e76df9405d91488dad0cfa7f8755bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e76df9405d91488dad0cfa7f8755bb1c already in progress: creating tablet
W20250625 07:52:13.333801 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b7139335c7124ab79bc2d0a251984199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7139335c7124ab79bc2d0a251984199 already in progress: creating tablet
W20250625 07:52:13.334048 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5f62b03db94d4ee3935866b5dd8bd53f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f62b03db94d4ee3935866b5dd8bd53f already in progress: creating tablet
W20250625 07:52:13.334261 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 834d516547b046288e58b7cde4c56e5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 834d516547b046288e58b7cde4c56e5a already in progress: creating tablet
I20250625 07:52:13.334926 1162 tablet_bootstrap.cc:492] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.335295 1163 tablet_bootstrap.cc:654] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.335565 1162 ts_tablet_manager.cc:1397] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
W20250625 07:52:13.335958 583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477): Couldn't send request to peer 00a3faeffaa4422ea73b98ee74768a4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:13.336911 1162 raft_consensus.cc:357] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.337661 1162 raft_consensus.cc:383] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.337818 1162 raft_consensus.cc:738] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.337895 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9861eea50d944b2c92be67c3c7c34582 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.338116 1162 consensus_queue.cc:260] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.339259 1162 ts_tablet_manager.cc:1428] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:13.339569 1162 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.341451 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e5f06c153744fb7a5104d5e5ac47575 with delete_type TABLET_DATA_DELETED (replaced by tablet be1f3fbffc954fdb8eb5105b5d383d0c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.342427 1162 tablet_bootstrap.cc:654] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.343928 1160 tablet_replica.cc:331] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.344116 1160 raft_consensus.cc:2241] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.344305 1160 raft_consensus.cc:2270] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.345360 1160 ts_tablet_manager.cc:1905] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.345372 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b662e261763d437fafa99ee7cade09e7" } modify_external_catalogs: true
I20250625 07:52:13.345587 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b662e261763d437fafa99ee7cade09e7" } modify_external_catalogs: true
I20250625 07:52:13.348119 886 tablet_service.cc:1515] Processing DeleteTablet for tablet be1f3fbffc954fdb8eb5105b5d383d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.352597 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.353832 1166 ts_tablet_manager.cc:1918] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.354128 1166 log.cc:1199] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9916612cb0234ec3a6df8f22c2b50e7c
I20250625 07:52:13.354642 1166 ts_tablet_manager.cc:1939] T 9916612cb0234ec3a6df8f22c2b50e7c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.355754 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9916612cb0234ec3a6df8f22c2b50e7c (table default.Table_9fdc22d5be4b43a4be699388015b9864 [id=7fbebf8268534597bd15d2bc70204cb7]) successfully deleted
I20250625 07:52:13.355830 1166 tablet_replica.cc:331] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.329432 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 6b3f2915b351407da08a9d6ba8ed402f (DEFAULT_TABLE table=default.Table_3497bef6f20d4203ab63ce8841a34b73 [id=1ab8d615f8a84fdeba96d8266c1d4555]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.356822 1166 raft_consensus.cc:2241] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.357066 1166 raft_consensus.cc:2270] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.357769 1166 ts_tablet_manager.cc:1905] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.358937 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_40651d9551314c82aab23007dc0c10d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.359699 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40651d9551314c82aab23007dc0c10d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.362066 1163 tablet_bootstrap.cc:492] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.362262 1163 ts_tablet_manager.cc:1397] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.006s
I20250625 07:52:13.363407 1163 raft_consensus.cc:357] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.363642 1163 raft_consensus.cc:383] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.363755 1163 raft_consensus.cc:738] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.363768 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d9d66546d7c42949dfe5db77f1057fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.365130 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e34ba3eabd8f4bafba2121af2e4ef24a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.365546 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49697a10cdac4c8ebb69f10f4c3437ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.365671 1158 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:13.365754 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e34ba3eabd8f4bafba2121af2e4ef24a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.366695 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.366959 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (DEFAULT_TABLE table=default.Table_0dd2a46a354a435497fe259c42f87353 [id=e441385dc1044575b17f64b06f14c648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.367359 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 9e800d4982fa45f4b931985c8b52719d from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.367498 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_15539c78bafe4ffa98fd75df3282bb82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.368077 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15539c78bafe4ffa98fd75df3282bb82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.368280 1163 consensus_queue.cc:260] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.369161 1160 ts_tablet_manager.cc:1918] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.369307 1160 log.cc:1199] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9861eea50d944b2c92be67c3c7c34582
I20250625 07:52:13.369725 1160 ts_tablet_manager.cc:1939] T 9861eea50d944b2c92be67c3c7c34582 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.372831 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "94ef72e8a4f54fd4994dcf4725337478" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:13.373176 915 raft_consensus.cc:2466] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f2cdaf87b9d4b2a919098bf65efd513 in term 0.
I20250625 07:52:13.373255 1163 ts_tablet_manager.cc:1428] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20250625 07:52:13.374011 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 94ef72e8a4f54fd4994dcf4725337478 with delete_type TABLET_DATA_DELETED (replaced by tablet 365ef91e7bac43e283602d3d036853ff) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.374418 1162 tablet_bootstrap.cc:492] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.374943 1162 ts_tablet_manager.cc:1397] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.011s
I20250625 07:52:13.375715 1162 raft_consensus.cc:357] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.375908 1162 raft_consensus.cc:383] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.378309 1162 raft_consensus.cc:738] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.378762 1162 consensus_queue.cc:260] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.383056 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.383417 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.386170 1163 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.386727 1160 tablet_replica.cc:331] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.386950 1160 raft_consensus.cc:2241] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.387149 1160 raft_consensus.cc:2270] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.387915 1163 tablet_bootstrap.cc:654] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.387964 1160 ts_tablet_manager.cc:1905] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:13.388968 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0452e73865964f709310fd22ac39a5f0 consensus state has no leader
I20250625 07:52:13.389446 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b3f2915b351407da08a9d6ba8ed402f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.392791 1166 ts_tablet_manager.cc:1918] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.392954 1166 log.cc:1199] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4e5f06c153744fb7a5104d5e5ac47575
I20250625 07:52:13.393235 755 tablet_service.cc:1468] Processing CreateTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 (DEFAULT_TABLE table=default.Table_889e54e43d5b47c888f9006ba7ff4f85 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.393381 1166 ts_tablet_manager.cc:1939] T 4e5f06c153744fb7a5104d5e5ac47575 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.393770 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b377c8111c9f4ac2ba05e65694d611b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.395838 752 tablet_service.cc:1468] Processing CreateTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (DEFAULT_TABLE table=default.Table_4ea5f6754f394000b657b1deee066ac7 [id=131a7d8cbeb34a0a8db53be305d31ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.396201 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29eaa8d93e5b4d1d8ed147b60c7da859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.397009 1160 ts_tablet_manager.cc:1918] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.397158 1160 log.cc:1199] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:13.397074 757 tablet_service.cc:1468] Processing CreateTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.397495 1160 ts_tablet_manager.cc:1939] T be1f3fbffc954fdb8eb5105b5d383d0c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.397532 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a2b94af59648239cfd1e5ce5e8b968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.398813 1160 tablet_replica.cc:331] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.398975 1160 raft_consensus.cc:2241] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.399164 1160 raft_consensus.cc:2270] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.399968 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d787caa22cfb4779bb6d87a5b5571157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.400576 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d787caa22cfb4779bb6d87a5b5571157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.400609 1163 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.400782 1163 ts_tablet_manager.cc:1397] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20250625 07:52:13.400988 758 tablet_service.cc:1468] Processing CreateTablet for tablet 5c28a064006145eea000e74da10d0794 (DEFAULT_TABLE table=default.Table_637a8e11bb45485398b914ec9dda3c77 [id=cbb5aafb8a9243d98c7d8578865c13b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.401314 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c28a064006145eea000e74da10d0794. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.401593 1163 raft_consensus.cc:357] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.401789 1163 raft_consensus.cc:383] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.401861 1163 raft_consensus.cc:738] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.402112 1163 consensus_queue.cc:260] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.402889 1163 ts_tablet_manager.cc:1428] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:13.403187 1163 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.404966 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6606a9559e6b47d28669083af5aee5bd (DEFAULT_TABLE table=default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.405216 1163 tablet_bootstrap.cc:654] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.405355 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6606a9559e6b47d28669083af5aee5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.406502 1160 ts_tablet_manager.cc:1905] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.406657 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9861eea50d944b2c92be67c3c7c34582 (table default.Table_dfdde9ef755044f7b3d9ab934ca4b231 [id=cc9a674f99634eae89fb70aad7f6ef99]) successfully deleted
I20250625 07:52:13.406891 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet be1f3fbffc954fdb8eb5105b5d383d0c (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]) successfully deleted
I20250625 07:52:13.409047 751 tablet_service.cc:1468] Processing CreateTablet for tablet de9ccb0a2d3846228acbc2769786a059 (DEFAULT_TABLE table=default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.409264 886 tablet_service.cc:1468] Processing CreateTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 (DEFAULT_TABLE table=default.Table_889e54e43d5b47c888f9006ba7ff4f85 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.409494 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ccb0a2d3846228acbc2769786a059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.409651 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b377c8111c9f4ac2ba05e65694d611b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.410207 885 tablet_service.cc:1468] Processing CreateTablet for tablet de9ccb0a2d3846228acbc2769786a059 (DEFAULT_TABLE table=default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.410560 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ccb0a2d3846228acbc2769786a059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.411162 884 tablet_service.cc:1468] Processing CreateTablet for tablet 6606a9559e6b47d28669083af5aee5bd (DEFAULT_TABLE table=default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.411504 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6606a9559e6b47d28669083af5aee5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.413462 1163 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.413609 1163 ts_tablet_manager.cc:1397] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250625 07:52:13.414368 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4e5f06c153744fb7a5104d5e5ac47575 (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]) successfully deleted
W20250625 07:52:13.414660 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 already in progress: creating tablet
I20250625 07:52:13.414705 1163 raft_consensus.cc:357] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.414892 1163 raft_consensus.cc:383] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.416203 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_66371f9b27d84da78425fe46ee94cdcf" } modify_external_catalogs: true
I20250625 07:52:13.416410 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_66371f9b27d84da78425fe46ee94cdcf" } modify_external_catalogs: true
W20250625 07:52:13.420301 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 23755f70822740e08ca6a5e5b5571458 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23755f70822740e08ca6a5e5b5571458 already in progress: creating tablet
I20250625 07:52:13.426517 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6382963863548398bed90f03b840806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.432335 1162 ts_tablet_manager.cc:1428] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20250625 07:52:13.433327 1162 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.434438 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 53c6c51a90754878977aeeaacf4773a6
I20250625 07:52:13.434823 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6606a9559e6b47d28669083af5aee5bd
I20250625 07:52:13.435551 1162 tablet_bootstrap.cc:654] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.435529 985 leader_election.cc:343] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9861eea50d944b2c92be67c3c7c34582
W20250625 07:52:13.435835 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 56315.8us
I20250625 07:52:13.437600 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 86ea9df068a94d9187f2f9620e371a39
I20250625 07:52:13.438832 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1e6c2988baa14cbc8c878f79e942880d
I20250625 07:52:13.442252 985 leader_election.cc:304] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513; no voters:
I20250625 07:52:13.444891 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_73a04c95dfa64a86bd47f8aacefad904"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.445578 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73a04c95dfa64a86bd47f8aacefad904 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.445683 1180 raft_consensus.cc:2802] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:13.445914 1180 raft_consensus.cc:491] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:13.447145 1162 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.447355 1162 ts_tablet_manager.cc:1397] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20250625 07:52:13.447834 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet de9ccb0a2d3846228acbc2769786a059
I20250625 07:52:13.448503 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.448217 1162 raft_consensus.cc:357] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.448669 1162 raft_consensus.cc:383] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.448836 1162 raft_consensus.cc:738] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.449018 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.449110 1162 consensus_queue.cc:260] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.449602 1162 ts_tablet_manager.cc:1428] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:13.449831 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.449908 1162 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.450294 755 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.450770 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.450994 1166 tablet_replica.cc:331] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.451220 1166 raft_consensus.cc:2241] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.451402 1166 raft_consensus.cc:2270] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:13.451644 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6606a9559e6b47d28669083af5aee5bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6606a9559e6b47d28669083af5aee5bd already in progress: creating tablet
W20250625 07:52:13.451942 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 86ea9df068a94d9187f2f9620e371a39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86ea9df068a94d9187f2f9620e371a39 already in progress: creating tablet
I20250625 07:52:13.452054 1166 ts_tablet_manager.cc:1905] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.452157 1162 tablet_bootstrap.cc:654] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.452956 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1e6c2988baa14cbc8c878f79e942880d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6c2988baa14cbc8c878f79e942880d already in progress: creating tablet
W20250625 07:52:13.453655 1096 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:38477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111)
I20250625 07:52:13.454408 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f63fdf2c8464c8fb58f412263d4e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.456092 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet de9ccb0a2d3846228acbc2769786a059 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de9ccb0a2d3846228acbc2769786a059 already in progress: creating tablet
I20250625 07:52:13.457793 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.459648 1166 ts_tablet_manager.cc:1918] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.459801 1166 log.cc:1199] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/37f7c060fb8648b9aaa766bbaa1b4ba7
I20250625 07:52:13.460170 1166 ts_tablet_manager.cc:1939] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.460402 1162 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.460530 1162 ts_tablet_manager.cc:1397] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
W20250625 07:52:13.460747 986 leader_election.cc:343] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9861eea50d944b2c92be67c3c7c34582
I20250625 07:52:13.461115 1180 raft_consensus.cc:3058] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:13.461344 1162 raft_consensus.cc:357] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.461555 1162 raft_consensus.cc:383] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.461651 1162 raft_consensus.cc:738] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.461730 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (table default.Table_b7677386183a479f9714604d7c2ab058 [id=e4b3ee29eafd4621811d8db11bdd65c4]) successfully deleted
I20250625 07:52:13.461886 1162 consensus_queue.cc:260] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:13.462038 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 53c6c51a90754878977aeeaacf4773a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53c6c51a90754878977aeeaacf4773a6 already in progress: creating tablet
I20250625 07:52:13.445585 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2908cac0353642cc92bc2152162de70e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.462392 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.462546 1162 ts_tablet_manager.cc:1428] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250625 07:52:13.462626 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2908cac0353642cc92bc2152162de70e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.462839 1162 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.464483 1162 tablet_bootstrap.cc:654] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.464821 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 (DEFAULT_TABLE table=default.Table_889e54e43d5b47c888f9006ba7ff4f85 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.466547 1022 tablet_service.cc:1468] Processing CreateTablet for tablet de9ccb0a2d3846228acbc2769786a059 (DEFAULT_TABLE table=default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.467108 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_16fdbb4827074529aaa4f9a805a01468"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.467626 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16fdbb4827074529aaa4f9a805a01468 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.468545 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.468652 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_236dd4e9c793494b8736319f9247a3da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.468822 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5c28a064006145eea000e74da10d0794 (DEFAULT_TABLE table=default.Table_637a8e11bb45485398b914ec9dda3c77 [id=cbb5aafb8a9243d98c7d8578865c13b9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:13.469172 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_236dd4e9c793494b8736319f9247a3da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.469161 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.469489 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.470268 588 master_service.cc:948] ReplaceTablet: received request to replace tablet b0d31993b4394cfd8832c93ab31b227b from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.470577 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 55d190e9d11c4589a82fcc7307e9eccb from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.472484 1162 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.472641 1162 ts_tablet_manager.cc:1397] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250625 07:52:13.472863 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 834d516547b046288e58b7cde4c56e5a with delete_type TABLET_DATA_DELETED (replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.472837 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_28a8c56dcb9b4bf093c733607ff00d69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.473369 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28a8c56dcb9b4bf093c733607ff00d69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.473354 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (DEFAULT_TABLE table=default.Table_4ea5f6754f394000b657b1deee066ac7 [id=131a7d8cbeb34a0a8db53be305d31ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.473508 1162 raft_consensus.cc:357] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.473698 1162 raft_consensus.cc:383] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.473789 1162 raft_consensus.cc:738] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.474025 1162 consensus_queue.cc:260] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.474448 1158 tablet_bootstrap.cc:654] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.474581 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 1222f2c1027b42c6ad9627359a93d678 (table default.Table_ca7c2bc86d9f4ad1896150b322f2d1b5 [id=5174ca174ab24d7695eb1fddc9414c74], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1222f2c1027b42c6ad9627359a93d678 consensus state has no leader
I20250625 07:52:13.454535 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6606a9559e6b47d28669083af5aee5bd (DEFAULT_TABLE table=default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.474413 1166 tablet_replica.cc:331] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.475270 1166 raft_consensus.cc:2241] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.475567 1166 raft_consensus.cc:2270] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.460893 986 leader_election.cc:304] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:13.476064 1162 ts_tablet_manager.cc:1428] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:13.476393 1162 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.476612 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ccb0a2d3846228acbc2769786a059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.477039 1161 ts_tablet_manager.cc:1918] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.477186 1161 log.cc:1199] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/bf0189f6651347a18663a94aa61ee267
I20250625 07:52:13.477643 1161 ts_tablet_manager.cc:1939] T bf0189f6651347a18663a94aa61ee267 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:13.477995 1162 tablet_bootstrap.cc:654] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.481042 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.481350 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a2b94af59648239cfd1e5ce5e8b968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.481845 1166 ts_tablet_manager.cc:1905] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.484875 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b377c8111c9f4ac2ba05e65694d611b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.488545 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c28a064006145eea000e74da10d0794. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.488627 1166 ts_tablet_manager.cc:1918] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.488762 1166 log.cc:1199] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/51279fb42ae8480b8511432cb4f808ab
I20250625 07:52:13.489239 1166 ts_tablet_manager.cc:1939] T 51279fb42ae8480b8511432cb4f808ab P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.490234 1166 tablet_replica.cc:331] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.490442 1166 raft_consensus.cc:2241] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.490600 1166 raft_consensus.cc:2270] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.491395 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29eaa8d93e5b4d1d8ed147b60c7da859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.494647 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 81976db0a2f94128a89842a12f067231 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.495347 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.495560 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.497435 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.497488 757 tablet_service.cc:1468] Processing CreateTablet for tablet 81976db0a2f94128a89842a12f067231 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.497889 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81976db0a2f94128a89842a12f067231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.498502 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 51279fb42ae8480b8511432cb4f808ab (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:13.500970 1163 raft_consensus.cc:738] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.501022 890 tablet_service.cc:1468] Processing CreateTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.501271 1163 consensus_queue.cc:260] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.501590 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a2b94af59648239cfd1e5ce5e8b968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.502153 1163 ts_tablet_manager.cc:1428] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.088s user 0.002s sys 0.001s
I20250625 07:52:13.502485 1163 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.503528 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.503665 895 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.503985 887 tablet_service.cc:1468] Processing CreateTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (DEFAULT_TABLE table=default.Table_4ea5f6754f394000b657b1deee066ac7 [id=131a7d8cbeb34a0a8db53be305d31ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.505735 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:13.505976 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5c28a064006145eea000e74da10d0794 (DEFAULT_TABLE table=default.Table_637a8e11bb45485398b914ec9dda3c77 [id=cbb5aafb8a9243d98c7d8578865c13b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.506342 588 catalog_manager.cc:6652] ReplaceTablet: tablet b0d31993b4394cfd8832c93ab31b227b deleted and replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:13.508246 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d31993b4394cfd8832c93ab31b227b with delete_type TABLET_DATA_DELETED (replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.520246 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.521207 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.521389 1166 ts_tablet_manager.cc:1905] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.521515 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.521925 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ab6d78ba7ead4a7f94152de5254f7bf0" } new_table_name: "default.Table_c589a7d832eb4e7fb74f00c8fcbaa680" modify_external_catalogs: true
I20250625 07:52:13.522722 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29eaa8d93e5b4d1d8ed147b60c7da859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.524430 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_15539c78bafe4ffa98fd75df3282bb82" } new_table_name: "default.Table_47373b3c8ede489ca51da1fea3723d35" modify_external_catalogs: true
I20250625 07:52:13.526291 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_889e54e43d5b47c888f9006ba7ff4f85" } new_table_name: "default.Table_55f91bee59d14d0a9744c1eccf96fc29" modify_external_catalogs: true
I20250625 07:52:13.527191 895 tablet_service.cc:1468] Processing CreateTablet for tablet 81976db0a2f94128a89842a12f067231 (DEFAULT_TABLE table=default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.527942 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4ea5f6754f394000b657b1deee066ac7" } new_table_name: "default.Table_635498945f814a89b120fc8a6d11e3d0" modify_external_catalogs: true
I20250625 07:52:13.528052 1166 ts_tablet_manager.cc:1918] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.528192 1166 log.cc:1199] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/834d516547b046288e58b7cde4c56e5a
I20250625 07:52:13.528609 1166 ts_tablet_manager.cc:1939] T 834d516547b046288e58b7cde4c56e5a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.529049 1163 tablet_bootstrap.cc:654] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.529417 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c28a064006145eea000e74da10d0794. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.531050 1166 tablet_replica.cc:331] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.531090 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 834d516547b046288e58b7cde4c56e5a (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:13.531334 1166 raft_consensus.cc:2241] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.531579 1166 raft_consensus.cc:2270] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.532830 891 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d31993b4394cfd8832c93ab31b227b with delete_type TABLET_DATA_DELETED (replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.533027 1166 ts_tablet_manager.cc:1905] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.534435 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81976db0a2f94128a89842a12f067231. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.535302 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0452e73865964f709310fd22ac39a5f0 consensus state has no leader
W20250625 07:52:13.540066 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a9945f38ef148caac59acaadcb2ef4c has no consensus state
W20250625 07:52:13.540580 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b377c8111c9f4ac2ba05e65694d611b9 consensus state has no leader
I20250625 07:52:13.546149 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet bf0189f6651347a18663a94aa61ee267 (table default.Table_0c3e0d58383c4ed0a3b898b5e73c8ff5 [id=900fc951961a4bfea2ae71d3b26fa7b0]) successfully deleted
W20250625 07:52:13.546247 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29eaa8d93e5b4d1d8ed147b60c7da859 consensus state has no leader
I20250625 07:52:13.546460 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d31993b4394cfd8832c93ab31b227b with delete_type TABLET_DATA_DELETED (replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.547715 1161 tablet_replica.cc:331] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:13.550510 1162 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.550699 1162 ts_tablet_manager.cc:1397] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.074s user 0.024s sys 0.028s
I20250625 07:52:13.551630 1166 ts_tablet_manager.cc:1918] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.551707 1162 raft_consensus.cc:357] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.551842 1166 log.cc:1199] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:13.552196 1162 raft_consensus.cc:383] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.552466 1162 raft_consensus.cc:738] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.552804 1162 consensus_queue.cc:260] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.553499 1162 ts_tablet_manager.cc:1428] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:13.553810 1162 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.556351 749 tablet_service.cc:1468] Processing CreateTablet for tablet a777f64425384be5bccc86b197626618 (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.556846 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc (DEFAULT_TABLE table=default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.556970 1166 ts_tablet_manager.cc:1939] T 95fce53f9a654fcaa8a52e5733c1b74f P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.557304 758 tablet_service.cc:1515] Processing DeleteTablet for tablet e90434606299495081d752df4761f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.556842 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777f64425384be5bccc86b197626618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.558324 1162 tablet_bootstrap.cc:654] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.559140 1163 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.558086 1160 ts_tablet_manager.cc:1918] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.559641 1163 ts_tablet_manager.cc:1397] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.003s
I20250625 07:52:13.560725 1163 raft_consensus.cc:357] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.561087 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4519b1fbf34ca8b33655f4d402aafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.561846 1163 raft_consensus.cc:383] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.561977 1163 raft_consensus.cc:738] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.562739 1163 consensus_queue.cc:260] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.563485 1163 ts_tablet_manager.cc:1428] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 07:52:13.559652 1160 log.cc:1199] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:13.565066 891 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc (DEFAULT_TABLE table=default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.565943 751 tablet_service.cc:1468] Processing CreateTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 (DEFAULT_TABLE table=default.Table_b3c9bae2008e48b199a1d0551ec879c7 [id=c2c95c5f1b914c829fa93e8955558fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.566255 1163 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.566295 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41e53efb6f342d2b1e4d642f7aaad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.566390 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4519b1fbf34ca8b33655f4d402aafc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.568984 752 tablet_service.cc:1468] Processing CreateTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 (DEFAULT_TABLE table=default.Table_d787caa22cfb4779bb6d87a5b5571157 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.569367 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edda0c3f60f64ed0b965eca52c5b3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.572768 889 tablet_service.cc:1468] Processing CreateTablet for tablet c65f257ae653488a8b49b0b523d2cf82 (DEFAULT_TABLE table=default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.573007 758 tablet_service.cc:1468] Processing CreateTablet for tablet c65f257ae653488a8b49b0b523d2cf82 (DEFAULT_TABLE table=default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.573143 886 tablet_service.cc:1468] Processing CreateTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 (DEFAULT_TABLE table=default.Table_b3c9bae2008e48b199a1d0551ec879c7 [id=c2c95c5f1b914c829fa93e8955558fa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.573400 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65f257ae653488a8b49b0b523d2cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.576670 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65f257ae653488a8b49b0b523d2cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.577680 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41e53efb6f342d2b1e4d642f7aaad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.577968 893 tablet_service.cc:1468] Processing CreateTablet for tablet a777f64425384be5bccc86b197626618 (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.578718 884 tablet_service.cc:1468] Processing CreateTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 (DEFAULT_TABLE table=default.Table_d787caa22cfb4779bb6d87a5b5571157 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.579051 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777f64425384be5bccc86b197626618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.580374 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c (DEFAULT_TABLE table=default.Table_47373b3c8ede489ca51da1fea3723d35 [id=836816eb909641daba87bdae20e0fd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.580765 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9945f38ef148caac59acaadcb2ef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.585369 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edda0c3f60f64ed0b965eca52c5b3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.586273 1160 ts_tablet_manager.cc:1939] T 94ef72e8a4f54fd4994dcf4725337478 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.586619 883 tablet_service.cc:1468] Processing CreateTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c (DEFAULT_TABLE table=default.Table_47373b3c8ede489ca51da1fea3723d35 [id=836816eb909641daba87bdae20e0fd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.587697 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9945f38ef148caac59acaadcb2ef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.587754 892 tablet_service.cc:1468] Processing CreateTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.588270 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100ef72ff14c24be92c21c6eb8a9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.589237 894 tablet_service.cc:1468] Processing CreateTablet for tablet 875fa1b46903452eaed1f2235d63dfdf (DEFAULT_TABLE table=default.Table_236dd4e9c793494b8736319f9247a3da [id=b7239cfa06b649ab946d863c06190160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.589841 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0024618293d74f559b04258f66616044 (DEFAULT_TABLE table=default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.586820 1163 tablet_bootstrap.cc:654] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.593544 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0f4d48075c0b46f3bfdb0c252203109f" } new_table_name: "default.Table_3b26428e16b14c4aaaf95f0f92b3670c" modify_external_catalogs: true
I20250625 07:52:13.593829 745 tablet_service.cc:1468] Processing CreateTablet for tablet 875fa1b46903452eaed1f2235d63dfdf (DEFAULT_TABLE table=default.Table_236dd4e9c793494b8736319f9247a3da [id=b7239cfa06b649ab946d863c06190160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.594252 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 875fa1b46903452eaed1f2235d63dfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.596172 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8b69c49b5fd54a4dacdae2d2792e230d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.596897 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b69c49b5fd54a4dacdae2d2792e230d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.597806 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 94ef72e8a4f54fd4994dcf4725337478 (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
W20250625 07:52:13.598301 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 86ea9df068a94d9187f2f9620e371a39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86ea9df068a94d9187f2f9620e371a39 already in progress: creating tablet
W20250625 07:52:13.598639 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet de9ccb0a2d3846228acbc2769786a059 because the tablet was not found. No further retry: Not found: Tablet not found: de9ccb0a2d3846228acbc2769786a059
W20250625 07:52:13.598978 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6606a9559e6b47d28669083af5aee5bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6606a9559e6b47d28669083af5aee5bd already in progress: creating tablet
W20250625 07:52:13.600414 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 53c6c51a90754878977aeeaacf4773a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53c6c51a90754878977aeeaacf4773a6 already in progress: creating tablet
W20250625 07:52:13.600883 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1e6c2988baa14cbc8c878f79e942880d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6c2988baa14cbc8c878f79e942880d already in progress: creating tablet
I20250625 07:52:13.600978 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 (DEFAULT_TABLE table=default.Table_b3c9bae2008e48b199a1d0551ec879c7 [id=c2c95c5f1b914c829fa93e8955558fa2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:13.601223 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b0d31993b4394cfd8832c93ab31b227b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0d31993b4394cfd8832c93ab31b227b already in progress: creating tablet
I20250625 07:52:13.601787 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6606a9559e6b47d28669083af5aee5bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.604915 881 tablet_service.cc:1468] Processing CreateTablet for tablet f84acd45845c4917b7dc8c5ad5232897 (DEFAULT_TABLE table=default.Table_e34ba3eabd8f4bafba2121af2e4ef24a [id=b0f32fb69f33447db7620299affa88c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:13.605053 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d9d66546d7c42949dfe5db77f1057fe consensus state has no leader
I20250625 07:52:13.605108 1024 tablet_service.cc:1468] Processing CreateTablet for tablet a777f64425384be5bccc86b197626618 (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.605441 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41e53efb6f342d2b1e4d642f7aaad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.605979 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a777f64425384be5bccc86b197626618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.609365 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 875fa1b46903452eaed1f2235d63dfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.614840 750 tablet_service.cc:1468] Processing CreateTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.617961 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c (DEFAULT_TABLE table=default.Table_47373b3c8ede489ca51da1fea3723d35 [id=836816eb909641daba87bdae20e0fd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.619081 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0024618293d74f559b04258f66616044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.619371 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81976db0a2f94128a89842a12f067231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.622493 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 875fa1b46903452eaed1f2235d63dfdf (DEFAULT_TABLE table=default.Table_236dd4e9c793494b8736319f9247a3da [id=b7239cfa06b649ab946d863c06190160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.623005 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc (DEFAULT_TABLE table=default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.622282 1019 tablet_service.cc:1468] Processing CreateTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 (DEFAULT_TABLE table=default.Table_d787caa22cfb4779bb6d87a5b5571157 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.623668 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84acd45845c4917b7dc8c5ad5232897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.625447 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 0024618293d74f559b04258f66616044 (DEFAULT_TABLE table=default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.623039 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c65f257ae653488a8b49b0b523d2cf82 (DEFAULT_TABLE table=default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.626487 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0024618293d74f559b04258f66616044 (DEFAULT_TABLE table=default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.627238 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100ef72ff14c24be92c21c6eb8a9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.627796 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5bc1b69194f54555909af8b886caf03d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.628499 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc1b69194f54555909af8b886caf03d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.628619 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9945f38ef148caac59acaadcb2ef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.630748 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_41d1f73c2d534e5496df588509888b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.631364 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d1f73c2d534e5496df588509888b0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.631572 1158 tablet_bootstrap.cc:492] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:13.631706 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0024618293d74f559b04258f66616044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.631733 1158 ts_tablet_manager.cc:1397] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.266s user 0.004s sys 0.022s
I20250625 07:52:13.632584 1158 raft_consensus.cc:357] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.632788 1158 raft_consensus.cc:383] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.632877 1158 raft_consensus.cc:738] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.634282 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f31530fc873b459298f1f673bc80cd7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.634835 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f31530fc873b459298f1f673bc80cd7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.635694 744 tablet_service.cc:1468] Processing CreateTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (DEFAULT_TABLE table=default.Table_98554e0797a84c44bfe6271791b5c975 [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.633971 747 tablet_service.cc:1468] Processing CreateTablet for tablet f84acd45845c4917b7dc8c5ad5232897 (DEFAULT_TABLE table=default.Table_e34ba3eabd8f4bafba2121af2e4ef24a [id=b0f32fb69f33447db7620299affa88c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.633062 1158 consensus_queue.cc:260] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.638464 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 875fa1b46903452eaed1f2235d63dfdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.640169 1248 raft_consensus.cc:2747] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:13.640470 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84acd45845c4917b7dc8c5ad5232897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.630893 1015 tablet_service.cc:1468] Processing CreateTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.645566 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3b26428e16b14c4aaaf95f0f92b3670c" } modify_external_catalogs: true
I20250625 07:52:13.645779 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3b26428e16b14c4aaaf95f0f92b3670c" } modify_external_catalogs: true
I20250625 07:52:13.646199 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3ed8b54f47094fea9a6e3de748e5ca57" } modify_external_catalogs: true
I20250625 07:52:13.646371 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_014f0dfc938843338f50edc64d69b63a" } modify_external_catalogs: true
I20250625 07:52:13.646426 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3ed8b54f47094fea9a6e3de748e5ca57" } modify_external_catalogs: true
I20250625 07:52:13.646533 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_014f0dfc938843338f50edc64d69b63a" } modify_external_catalogs: true
I20250625 07:52:13.646639 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_be2a902089674d54ad4be0f8d749236f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.646994 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e407dc6ea5d247b1a9c26729116da0e2" } modify_external_catalogs: true
I20250625 07:52:13.647159 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e407dc6ea5d247b1a9c26729116da0e2" } modify_external_catalogs: true
W20250625 07:52:13.647192 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be2a902089674d54ad4be0f8d749236f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.647929 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.643458 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 188355us
I20250625 07:52:13.654918 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (DEFAULT_TABLE table=default.Table_98554e0797a84c44bfe6271791b5c975 [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.655360 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f84acd45845c4917b7dc8c5ad5232897 (DEFAULT_TABLE table=default.Table_e34ba3eabd8f4bafba2121af2e4ef24a [id=b0f32fb69f33447db7620299affa88c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.659361 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4519b1fbf34ca8b33655f4d402aafc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.669243 986 leader_election.cc:343] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:13.673513 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_73a04c95dfa64a86bd47f8aacefad904" } modify_external_catalogs: true
I20250625 07:52:13.673775 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_73a04c95dfa64a86bd47f8aacefad904" } modify_external_catalogs: true
I20250625 07:52:13.677390 1163 tablet_bootstrap.cc:492] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.677580 1163 ts_tablet_manager.cc:1397] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.114s user 0.009s sys 0.018s
I20250625 07:52:13.678506 1163 raft_consensus.cc:357] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.678694 1163 raft_consensus.cc:383] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.678788 1163 raft_consensus.cc:738] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.678803 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e14b60ee7a843e5bb878148d8fd2e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.678984 1248 raft_consensus.cc:491] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.679019 1163 consensus_queue.cc:260] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.681034 1248 raft_consensus.cc:513] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.682757 1163 ts_tablet_manager.cc:1428] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:13.683136 1163 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.687619 880 tablet_service.cc:1468] Processing CreateTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (DEFAULT_TABLE table=default.Table_98554e0797a84c44bfe6271791b5c975 [id=58cee91e0a014b2ea44ba7a6538e667b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.688735 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edda0c3f60f64ed0b965eca52c5b3591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.681718 1180 raft_consensus.cc:513] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.693923 1163 tablet_bootstrap.cc:654] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.693990 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65f257ae653488a8b49b0b523d2cf82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.695079 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3b0a3a4ca089491d8571b826333d5d9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.695751 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b0a3a4ca089491d8571b826333d5d9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.697564 1248 leader_election.cc:290] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.697783 1161 raft_consensus.cc:2241] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.698324 1161 raft_consensus.cc:2270] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.700111 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.702265 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.705168 986 leader_election.cc:343] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 37f7c060fb8648b9aaa766bbaa1b4ba7
W20250625 07:52:13.705426 985 leader_election.cc:343] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 37f7c060fb8648b9aaa766bbaa1b4ba7
W20250625 07:52:13.711067 986 leader_election.cc:343] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:13.716434 985 leader_election.cc:304] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:13.717908 1252 raft_consensus.cc:2747] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:13.719455 1248 raft_consensus.cc:491] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.730731 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2d8d3300befc4f419070ae84725ee96a (DEFAULT_TABLE table=default.Table_8b69c49b5fd54a4dacdae2d2792e230d [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.731223 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8d3300befc4f419070ae84725ee96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.739312 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4d7c95ebabf74ee1a260a5996137c0a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.740007 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d7c95ebabf74ee1a260a5996137c0a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.740089 1167 raft_consensus.cc:491] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.740875 1167 raft_consensus.cc:513] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.742372 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 834d516547b046288e58b7cde4c56e5a with delete_type TABLET_DATA_DELETED (replaced by tablet c7a2b94af59648239cfd1e5ce5e8b968) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.742615 1160 tablet_replica.cc:331] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.742846 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dbf9b53cf6f843c1b02c9793e9bc0741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.743018 1160 raft_consensus.cc:2241] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.743191 1160 raft_consensus.cc:2270] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:13.743417 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.746978 1180 leader_election.cc:290] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.747061 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fa9514049a2142ca95833a0e0a0bf14b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.747301 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0024618293d74f559b04258f66616044. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.747329 1158 ts_tablet_manager.cc:1428] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.115s user 0.010s sys 0.025s
W20250625 07:52:13.747571 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa9514049a2142ca95833a0e0a0bf14b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.747946 1158 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:13.748432 1161 ts_tablet_manager.cc:1905] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.751788 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100ef72ff14c24be92c21c6eb8a9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.752494 1163 tablet_bootstrap.cc:492] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.752714 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84acd45845c4917b7dc8c5ad5232897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.752708 1167 leader_election.cc:290] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:13.743345 851 leader_election.cc:343] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 51279fb42ae8480b8511432cb4f808ab
W20250625 07:52:13.758311 985 leader_election.cc:343] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:13.758778 1160 ts_tablet_manager.cc:1905] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.759495 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.760864 1248 raft_consensus.cc:513] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.761584 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 95fce53f9a654fcaa8a52e5733c1b74f (table default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]) successfully deleted
I20250625 07:52:13.762539 1166 tablet_replica.cc:331] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:13.762717 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b0d31993b4394cfd8832c93ab31b227b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0d31993b4394cfd8832c93ab31b227b already in progress: creating tablet
I20250625 07:52:13.762814 1166 raft_consensus.cc:2241] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.762995 1166 raft_consensus.cc:2270] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.763883 1166 ts_tablet_manager.cc:1905] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.765307 985 leader_election.cc:304] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:13.765545 985 raft_consensus.cc:2684] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:13.765782 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51279fb42ae8480b8511432cb4f808ab" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:13.767903 1163 ts_tablet_manager.cc:1397] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.017s
I20250625 07:52:13.769028 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.768380 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2d8d3300befc4f419070ae84725ee96a (DEFAULT_TABLE table=default.Table_8b69c49b5fd54a4dacdae2d2792e230d [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.771540 1163 raft_consensus.cc:357] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:13.773114 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_41e55514978543459d3cbec553136e88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.773249 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0024618293d74f559b04258f66616044
I20250625 07:52:13.773573 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6b3488a9fb19452dac3241170dd2fee2
W20250625 07:52:13.773702 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41e55514978543459d3cbec553136e88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.774003 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ac4400688af8429ea02aeec0e8d6b03b" } new_table_name: "default.Table_6960d95c6d2f46529495f6a7be9a0d43" modify_external_catalogs: true
I20250625 07:52:13.774328 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3497bef6f20d4203ab63ce8841a34b73" } new_table_name: "default.Table_8b2d59eebc0448e78ca54efc7537619e" modify_external_catalogs: true
I20250625 07:52:13.775039 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98554e0797a84c44bfe6271791b5c975" } new_table_name: "default.Table_12188272bb8e4c26bb94db8721d37c26" modify_external_catalogs: true
I20250625 07:52:13.776484 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:13.776635 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ebb943df93284913b6a520e099399322
I20250625 07:52:13.777263 588 master_service.cc:948] ReplaceTablet: received request to replace tablet ebb943df93284913b6a520e099399322 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.777221 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_637a8e11bb45485398b914ec9dda3c77" } new_table_name: "default.Table_9bbce7454a884554b31faffc79a33b99" modify_external_catalogs: true
I20250625 07:52:13.778244 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 81976db0a2f94128a89842a12f067231
W20250625 07:52:13.781087 986 leader_election.cc:343] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9916612cb0234ec3a6df8f22c2b50e7c
I20250625 07:52:13.781339 1163 raft_consensus.cc:383] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.781455 1248 leader_election.cc:290] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.781468 1163 raft_consensus.cc:738] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:13.781791 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 45614.5us
I20250625 07:52:13.781914 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.782423 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e90434606299495081d752df4761f9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.782905 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.783012 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 2d8d3300befc4f419070ae84725ee96a (DEFAULT_TABLE table=default.Table_8b69c49b5fd54a4dacdae2d2792e230d [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.782645 1163 consensus_queue.cc:260] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:13.783586 1163 ts_tablet_manager.cc:1428] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250625 07:52:13.784304 1163 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.782161 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.791378 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.791611 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.793944 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.794118 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8d3300befc4f419070ae84725ee96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.801091 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d8d3300befc4f419070ae84725ee96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.801241 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.802577 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 94ef72e8a4f54fd4994dcf4725337478 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.803346 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.805807 1162 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.806021 1162 ts_tablet_manager.cc:1397] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.252s user 0.031s sys 0.078s
I20250625 07:52:13.807376 1162 raft_consensus.cc:357] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.807763 1162 raft_consensus.cc:383] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.807907 1162 raft_consensus.cc:738] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.808238 1162 consensus_queue.cc:260] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.808735 588 master_service.cc:948] ReplaceTablet: received request to replace tablet aec412a15b964060a24631e7ebcf1ef7 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.811358 1162 ts_tablet_manager.cc:1428] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:13.811844 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4efdc728e9ca49549585528cc7eb22ee" } new_table_name: "default.Table_aac2883a6c934d2aa9de60cf6f345fad" modify_external_catalogs: true
I20250625 07:52:13.812225 1162 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.816156 1162 tablet_bootstrap.cc:654] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.817778 1188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:13.818148 1188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:13.820849 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_56e7b5805a334eb386aa31e60ece4da8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.821594 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56e7b5805a334eb386aa31e60ece4da8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.825157 1163 tablet_bootstrap.cc:654] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:13.827114 850 leader_election.cc:343] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:13.827203 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.828092 850 leader_election.cc:304] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:13.829398 1167 raft_consensus.cc:2747] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:13.829864 985 leader_election.cc:343] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9916612cb0234ec3a6df8f22c2b50e7c
W20250625 07:52:13.830714 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 49723.4us
I20250625 07:52:13.831960 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.832304 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.832397 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.832808 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.833047 1162 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.833225 1162 ts_tablet_manager.cc:1397] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
W20250625 07:52:13.833303 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b1eaf0266c04873b185c1c0c2da201c consensus state has no leader
I20250625 07:52:13.834031 1162 raft_consensus.cc:357] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.834247 1162 raft_consensus.cc:383] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.834333 1162 raft_consensus.cc:738] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:13.835477 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b3f2915b351407da08a9d6ba8ed402f consensus state has no leader
W20250625 07:52:13.835870 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c28a064006145eea000e74da10d0794 consensus state has no leader
W20250625 07:52:13.836375 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe consensus state has no leader
I20250625 07:52:13.840472 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.840716 1162 consensus_queue.cc:260] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.841264 1163 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.841424 1163 ts_tablet_manager.cc:1397] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.002s
I20250625 07:52:13.842063 1162 ts_tablet_manager.cc:1428] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250625 07:52:13.842276 1163 raft_consensus.cc:357] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.842401 1162 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.842463 1163 raft_consensus.cc:383] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.842558 1163 raft_consensus.cc:738] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.842774 1163 consensus_queue.cc:260] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.843263 1163 ts_tablet_manager.cc:1428] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:13.844012 1162 tablet_bootstrap.cc:654] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.844445 750 tablet_service.cc:1468] Processing CreateTablet for tablet b5f917cfd2384a72920847f8b89358b9 (DEFAULT_TABLE table=default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.844643 1160 ts_tablet_manager.cc:1918] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.845533 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f917cfd2384a72920847f8b89358b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.845769 1160 log.cc:1199] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/834d516547b046288e58b7cde4c56e5a
I20250625 07:52:13.846359 1160 ts_tablet_manager.cc:1939] T 834d516547b046288e58b7cde4c56e5a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:13.847651 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 51279fb42ae8480b8511432cb4f808ab from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.844908 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.849566 749 tablet_service.cc:1468] Processing CreateTablet for tablet 21658216da3544a2b8b8f39667782ffb (DEFAULT_TABLE table=default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.851022 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21658216da3544a2b8b8f39667782ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.851116 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet aec412a15b964060a24631e7ebcf1ef7
I20250625 07:52:13.851624 588 catalog_manager.cc:6652] ReplaceTablet: tablet aec412a15b964060a24631e7ebcf1ef7 deleted and replaced by tablet 10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:13.850622 746 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:13.854127 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32bd7cdef96946e88683e457c2f26c9b consensus state has no leader
I20250625 07:52:13.853718 745 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.850011 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 (DEFAULT_TABLE table=default.Table_f31530fc873b459298f1f673bc80cd7f [id=d24d9520ff924056a204dfd910f4dbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.860903 985 leader_election.cc:304] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:13.854790 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad78944352e4274acd8d9889b1d32b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.861792 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.862471 886 tablet_service.cc:1468] Processing CreateTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 (DEFAULT_TABLE table=default.Table_f31530fc873b459298f1f673bc80cd7f [id=d24d9520ff924056a204dfd910f4dbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.863175 881 tablet_service.cc:1468] Processing CreateTablet for tablet b5f917cfd2384a72920847f8b89358b9 (DEFAULT_TABLE table=default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.863219 882 tablet_service.cc:1468] Processing CreateTablet for tablet 21658216da3544a2b8b8f39667782ffb (DEFAULT_TABLE table=default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.863447 890 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.871529 745 tablet_service.cc:1468] Processing CreateTablet for tablet 7fd497aa0b8c4838906524f4423c3756 (DEFAULT_TABLE table=default.Table_41e55514978543459d3cbec553136e88 [id=368053320f3844fbbd5b935b9842499a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.871840 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7fd497aa0b8c4838906524f4423c3756 (DEFAULT_TABLE table=default.Table_41e55514978543459d3cbec553136e88 [id=368053320f3844fbbd5b935b9842499a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.871999 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd497aa0b8c4838906524f4423c3756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.873741 1163 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.876093 880 tablet_service.cc:1468] Processing CreateTablet for tablet a7cb2bcdd07c42e680eb4f155212423f (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.877493 1166 ts_tablet_manager.cc:1918] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.877642 1166 log.cc:1199] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e90434606299495081d752df4761f9f8
I20250625 07:52:13.878006 1166 ts_tablet_manager.cc:1939] T e90434606299495081d752df4761f9f8 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.878700 1163 tablet_bootstrap.cc:654] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.879424 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7f196473315441c6bf8b5173ed46f833 (DEFAULT_TABLE table=default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.880856 1158 tablet_bootstrap.cc:654] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.881090 1248 raft_consensus.cc:2747] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:13.881443 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 46135.1us
I20250625 07:52:13.885174 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 21658216da3544a2b8b8f39667782ffb (DEFAULT_TABLE table=default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.885609 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21658216da3544a2b8b8f39667782ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.885710 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 753404a0a77c46e6a90fbacb6cdfdbe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.886246 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:13.886746 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad78944352e4274acd8d9889b1d32b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.889658 893 tablet_service.cc:1468] Processing CreateTablet for tablet a3670cadfa2841949705c65c9b6c0927 (DEFAULT_TABLE table=default.Table_4d7c95ebabf74ee1a260a5996137c0a0 [id=dd2e80e6531a4e3f9be3b2fc19aec035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.890015 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3670cadfa2841949705c65c9b6c0927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.896596 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 834d516547b046288e58b7cde4c56e5a (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:13.891881 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b5f917cfd2384a72920847f8b89358b9 (DEFAULT_TABLE table=default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.900722 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f917cfd2384a72920847f8b89358b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.900882 1166 tablet_replica.cc:331] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.901127 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b1eaf0266c04873b185c1c0c2da201c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:13.892491 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 7f196473315441c6bf8b5173ed46f833 (DEFAULT_TABLE table=default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.897874 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 (DEFAULT_TABLE table=default.Table_f31530fc873b459298f1f673bc80cd7f [id=d24d9520ff924056a204dfd910f4dbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.897987 1248 raft_consensus.cc:491] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:13.901114 1166 raft_consensus.cc:2241] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.902616 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7cb2bcdd07c42e680eb4f155212423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.902719 1166 raft_consensus.cc:2270] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.903697 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e90434606299495081d752df4761f9f8 (table default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]) successfully deleted
W20250625 07:52:13.903972 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 81976db0a2f94128a89842a12f067231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81976db0a2f94128a89842a12f067231 already in progress: creating tablet
W20250625 07:52:13.904187 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7d9d66546d7c42949dfe5db77f1057fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9d66546d7c42949dfe5db77f1057fe already in progress: creating tablet
W20250625 07:52:13.904371 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ebb943df93284913b6a520e099399322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebb943df93284913b6a520e099399322 already in progress: creating tablet
W20250625 07:52:13.904552 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6b3488a9fb19452dac3241170dd2fee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b3488a9fb19452dac3241170dd2fee2 already in progress: creating tablet
I20250625 07:52:13.904501 747 tablet_service.cc:1468] Processing CreateTablet for tablet a3670cadfa2841949705c65c9b6c0927 (DEFAULT_TABLE table=default.Table_4d7c95ebabf74ee1a260a5996137c0a0 [id=dd2e80e6531a4e3f9be3b2fc19aec035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.904927 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f196473315441c6bf8b5173ed46f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.904896 746 tablet_service.cc:1468] Processing CreateTablet for tablet 7f196473315441c6bf8b5173ed46f833 (DEFAULT_TABLE table=default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.905004 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3670cadfa2841949705c65c9b6c0927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.905004 751 tablet_service.cc:1468] Processing CreateTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 (DEFAULT_TABLE table=default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.905323 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f196473315441c6bf8b5173ed46f833. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.905449 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0024618293d74f559b04258f66616044 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0024618293d74f559b04258f66616044 already in progress: creating tablet
I20250625 07:52:13.905684 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651c74bf9a9b47d6ab5a875a5c304783. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.906824 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5f917cfd2384a72920847f8b89358b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.909011 757 tablet_service.cc:1468] Processing CreateTablet for tablet ba24bbd0c4694930978a7750f07601d9 (DEFAULT_TABLE table=default.Table_3b0a3a4ca089491d8571b826333d5d9e [id=30de507c3fb344ecb071e9def7dcf119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.909389 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba24bbd0c4694930978a7750f07601d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.909407 884 tablet_service.cc:1468] Processing CreateTablet for tablet ba24bbd0c4694930978a7750f07601d9 (DEFAULT_TABLE table=default.Table_3b0a3a4ca089491d8571b826333d5d9e [id=30de507c3fb344ecb071e9def7dcf119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.909756 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba24bbd0c4694930978a7750f07601d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.917443 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd497aa0b8c4838906524f4423c3756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.918251 1163 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:13.918303 750 tablet_service.cc:1468] Processing CreateTablet for tablet a7cb2bcdd07c42e680eb4f155212423f (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.918435 1163 ts_tablet_manager.cc:1397] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250625 07:52:13.918730 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7cb2bcdd07c42e680eb4f155212423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.919293 1163 raft_consensus.cc:357] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.919518 1163 raft_consensus.cc:383] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.919593 1163 raft_consensus.cc:738] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.919868 1163 consensus_queue.cc:260] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.924791 1163 ts_tablet_manager.cc:1428] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:13.925478 1163 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:13.924793 893 tablet_service.cc:1468] Processing CreateTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 (DEFAULT_TABLE table=default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.928658 1163 tablet_bootstrap.cc:654] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.929003 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651c74bf9a9b47d6ab5a875a5c304783. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.929679 1162 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.929831 1162 ts_tablet_manager.cc:1397] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.035s
I20250625 07:52:13.930647 1162 raft_consensus.cc:357] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.930862 1162 raft_consensus.cc:383] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.930964 1162 raft_consensus.cc:738] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.931289 1162 consensus_queue.cc:260] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.931831 1162 ts_tablet_manager.cc:1428] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:13.932204 1162 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.934301 1162 tablet_bootstrap.cc:654] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.935561 1160 tablet_replica.cc:331] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:13.936105 1160 raft_consensus.cc:2241] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.938179 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f196473315441c6bf8b5173ed46f833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.938628 1162 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.938865 1162 ts_tablet_manager.cc:1397] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250625 07:52:13.938882 1160 raft_consensus.cc:2270] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.939739 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a7cb2bcdd07c42e680eb4f155212423f (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.892076 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:13.940279 1162 raft_consensus.cc:357] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.940601 1162 raft_consensus.cc:383] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.940730 1162 raft_consensus.cc:738] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.941072 1162 consensus_queue.cc:260] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.942459 1162 ts_tablet_manager.cc:1428] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:13.892261 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 7fd497aa0b8c4838906524f4423c3756 (DEFAULT_TABLE table=default.Table_41e55514978543459d3cbec553136e88 [id=368053320f3844fbbd5b935b9842499a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.944036 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ad78944352e4274acd8d9889b1d32b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.944723 1248 raft_consensus.cc:513] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:13.944854 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:13.945398 589 catalog_manager.cc:6652] ReplaceTablet: tablet 6b1eaf0266c04873b185c1c0c2da201c deleted and replaced by tablet 445bfed532004108a12c8fd78370c1cb
I20250625 07:52:13.947503 757 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.947423 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b14bab363cb9435f86290aa16ed47a6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.947537 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.947731 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:13.948045 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b14bab363cb9435f86290aa16ed47a6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:13.948129 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.951813 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a41e6c6e1d6342a498f4f79c95c7ee52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.952474 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a41e6c6e1d6342a498f4f79c95c7ee52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.955251 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a3670cadfa2841949705c65c9b6c0927 (DEFAULT_TABLE table=default.Table_4d7c95ebabf74ee1a260a5996137c0a0 [id=dd2e80e6531a4e3f9be3b2fc19aec035]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.958359 1166 ts_tablet_manager.cc:1905] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.964802 1162 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.964828 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.966399 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fd497aa0b8c4838906524f4423c3756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.967167 1162 tablet_bootstrap.cc:654] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.968753 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3670cadfa2841949705c65c9b6c0927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.975262 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ba24bbd0c4694930978a7750f07601d9 (DEFAULT_TABLE table=default.Table_3b0a3a4ca089491d8571b826333d5d9e [id=30de507c3fb344ecb071e9def7dcf119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.975615 750 tablet_service.cc:1468] Processing CreateTablet for tablet 10c7f4b4187c428bbec136c85219da4d (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.975759 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7cb2bcdd07c42e680eb4f155212423f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.976123 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7f4b4187c428bbec136c85219da4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.976349 1166 ts_tablet_manager.cc:1918] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.976543 1166 log.cc:1199] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:13.977020 1166 ts_tablet_manager.cc:1939] T 6fc7c0feb0e04ef28c323323e3007efc P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:13.978094 1166 tablet_replica.cc:331] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:13.978325 1166 raft_consensus.cc:2241] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:13.978503 1166 raft_consensus.cc:2270] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:13.979655 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6fc7c0feb0e04ef28c323323e3007efc (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:13.979724 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2" } modify_external_catalogs: true
I20250625 07:52:13.979918 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2" } modify_external_catalogs: true
I20250625 07:52:13.980340 746 tablet_service.cc:1468] Processing CreateTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 (DEFAULT_TABLE table=default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.980782 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802c22a6346442c88abbeb4b9ef20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:13.980674 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6e28f36e9e034b489bf92ac42b5f0e28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:13.981434 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e28f36e9e034b489bf92ac42b5f0e28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:13.983554 1162 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:13.983706 1162 ts_tablet_manager.cc:1397] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.002s
I20250625 07:52:13.984557 1162 raft_consensus.cc:357] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.984803 1162 raft_consensus.cc:383] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:13.984921 1162 raft_consensus.cc:738] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:13.985016 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_53870f55fcc5414ab6e8ccb92db24344" } modify_external_catalogs: true
I20250625 07:52:13.985291 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_53870f55fcc5414ab6e8ccb92db24344" } modify_external_catalogs: true
I20250625 07:52:13.985231 1162 consensus_queue.cc:260] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:13.985816 1162 ts_tablet_manager.cc:1428] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:13.986635 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c589a7d832eb4e7fb74f00c8fcbaa680" } modify_external_catalogs: true
I20250625 07:52:13.986802 1162 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:13.986784 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c589a7d832eb4e7fb74f00c8fcbaa680" } modify_external_catalogs: true
I20250625 07:52:13.987340 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7c4e84ad77a74e08a1c7aab848d552b7" } modify_external_catalogs: true
I20250625 07:52:13.987412 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5bc1b69194f54555909af8b886caf03d" } modify_external_catalogs: true
I20250625 07:52:13.987519 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7c4e84ad77a74e08a1c7aab848d552b7" } modify_external_catalogs: true
I20250625 07:52:13.987555 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5bc1b69194f54555909af8b886caf03d" } modify_external_catalogs: true
I20250625 07:52:13.988621 1162 tablet_bootstrap.cc:654] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:13.988814 1248 leader_election.cc:290] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:13.990346 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba24bbd0c4694930978a7750f07601d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:13.991189 986 leader_election.cc:343] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:13.989679 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 (DEFAULT_TABLE table=default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:13.991807 1166 ts_tablet_manager.cc:1905] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:13.993544 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:13.995931 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:13.999640 1166 ts_tablet_manager.cc:1918] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:13.999826 1166 log.cc:1199] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:14.000226 1166 ts_tablet_manager.cc:1939] T b09dc0b8a496402b82f98daed9af0d12 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.001483 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b09dc0b8a496402b82f98daed9af0d12 (table default.Table_c931105d39a449fdbdc73dc16830a6a4 [id=7f77bfd2215a491d8fba698477109409]) successfully deleted
I20250625 07:52:14.001592 1166 tablet_replica.cc:331] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.001809 1166 raft_consensus.cc:2241] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:14.001912 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet aec412a15b964060a24631e7ebcf1ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aec412a15b964060a24631e7ebcf1ef7 already in progress: creating tablet
I20250625 07:52:14.001991 1166 raft_consensus.cc:2270] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.002704 1166 ts_tablet_manager.cc:1905] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.007119 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651c74bf9a9b47d6ab5a875a5c304783. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.009898 1162 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.010124 1162 ts_tablet_manager.cc:1397] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250625 07:52:14.011171 1162 raft_consensus.cc:357] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.011431 1162 raft_consensus.cc:383] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.011513 1162 raft_consensus.cc:738] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.011821 1162 consensus_queue.cc:260] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.012457 1162 ts_tablet_manager.cc:1428] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:14.013967 1162 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:14.017146 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.020279 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21658216da3544a2b8b8f39667782ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.022550 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 (DEFAULT_TABLE table=default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.023267 750 tablet_service.cc:1468] Processing CreateTablet for tablet d4c744fef0e24cb39c4bb0787adb1cbc (DEFAULT_TABLE table=default.Table_b14bab363cb9435f86290aa16ed47a6c [id=29e4ffaed4f148b6b10e38231435c071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.023694 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c744fef0e24cb39c4bb0787adb1cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.025776 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 09b3830bd506443ab6f06ddd3c6b9974
I20250625 07:52:14.026767 1160 ts_tablet_manager.cc:1905] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.027208 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0452e73865964f709310fd22ac39a5f0
I20250625 07:52:14.028218 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 21658216da3544a2b8b8f39667782ffb
I20250625 07:52:14.032543 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802c22a6346442c88abbeb4b9ef20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.037262 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.040300 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 10c7f4b4187c428bbec136c85219da4d (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:14.045486 985 leader_election.cc:343] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:14.047360 1161 ts_tablet_manager.cc:1918] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.048357 1161 log.cc:1199] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/94ef72e8a4f54fd4994dcf4725337478
I20250625 07:52:14.048827 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.049398 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7f4b4187c428bbec136c85219da4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.049623 884 tablet_service.cc:1468] Processing CreateTablet for tablet d4c744fef0e24cb39c4bb0787adb1cbc (DEFAULT_TABLE table=default.Table_b14bab363cb9435f86290aa16ed47a6c [id=29e4ffaed4f148b6b10e38231435c071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.050187 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c744fef0e24cb39c4bb0787adb1cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.050352 985 leader_election.cc:304] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.053097 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.053534 889 tablet_service.cc:1468] Processing CreateTablet for tablet 31942f0998bb4118a91f476325851d5c (DEFAULT_TABLE table=default.Table_a41e6c6e1d6342a498f4f79c95c7ee52 [id=7686e223b6fb47e79ecb2257550a64b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.061349 1248 raft_consensus.cc:2747] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.067368 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_79f17dafcee94151a9c0ee944c8037f9" } new_table_name: "default.Table_1c386358ebca4d0c9f49fc20d6278f3e" modify_external_catalogs: true
I20250625 07:52:14.067584 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.067963 751 tablet_service.cc:1468] Processing CreateTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.068449 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08bb47266e8b4a0fa5ec613560b8549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.069067 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0bed2d3626248cfa65f9de3024542fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.069618 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0bed2d3626248cfa65f9de3024542fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.070806 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4bd7f1d95c4a4f62815742322b59570c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.071224 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f31530fc873b459298f1f673bc80cd7f" } new_table_name: "default.Table_bb291540701c4a4fbb5f3fa31fc4b684" modify_external_catalogs: true
I20250625 07:52:14.071708 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.071872 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:14.071324 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd7f1d95c4a4f62815742322b59570c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.072809 1166 ts_tablet_manager.cc:1918] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.072757 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c24058d938ab46a3a19827cf537af456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.072930 1166 log.cc:1199] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a558e0a5403d4489892dfce9587e090c
I20250625 07:52:14.072969 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_43f3b349ae154f228f90a77e3c33c46d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.073297 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c24058d938ab46a3a19827cf537af456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.073489 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f3b349ae154f228f90a77e3c33c46d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.074749 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3f4a64baf13c4985b561eae647c888bd" } new_table_name: "default.Table_1e843117005043efadef968ca11c482f" modify_external_catalogs: true
I20250625 07:52:14.074853 1166 ts_tablet_manager.cc:1939] T a558e0a5403d4489892dfce9587e090c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.074888 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31942f0998bb4118a91f476325851d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.075259 747 tablet_service.cc:1468] Processing CreateTablet for tablet 31942f0998bb4118a91f476325851d5c (DEFAULT_TABLE table=default.Table_a41e6c6e1d6342a498f4f79c95c7ee52 [id=7686e223b6fb47e79ecb2257550a64b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.053735 880 tablet_service.cc:1468] Processing CreateTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 (DEFAULT_TABLE table=default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.076004 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31942f0998bb4118a91f476325851d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.076097 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a558e0a5403d4489892dfce9587e090c (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
I20250625 07:52:14.076257 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802c22a6346442c88abbeb4b9ef20c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.055119 885 tablet_service.cc:1468] Processing CreateTablet for tablet 445bfed532004108a12c8fd78370c1cb (DEFAULT_TABLE table=default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.080780 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.080957 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445bfed532004108a12c8fd78370c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.081043 1166 tablet_replica.cc:331] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:14.081215 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6b1eaf0266c04873b185c1c0c2da201c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b1eaf0266c04873b185c1c0c2da201c already in progress: creating tablet
I20250625 07:52:14.081246 1166 raft_consensus.cc:2241] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.081459 1166 raft_consensus.cc:2270] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.081604 745 tablet_service.cc:1468] Processing CreateTablet for tablet 445bfed532004108a12c8fd78370c1cb (DEFAULT_TABLE table=default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.081967 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445bfed532004108a12c8fd78370c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.083045 1166 ts_tablet_manager.cc:1905] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.083863 1162 tablet_bootstrap.cc:654] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.056011 881 tablet_service.cc:1468] Processing CreateTablet for tablet 10c7f4b4187c428bbec136c85219da4d (DEFAULT_TABLE table=default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.085016 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7f4b4187c428bbec136c85219da4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.057027 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.097254 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--webserver_interface=127.0.104.62
--webserver_port=39255
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 07:52:14.064476 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.064910 882 tablet_service.cc:1468] Processing CreateTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.098536 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08bb47266e8b4a0fa5ec613560b8549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.065394 1160 ts_tablet_manager.cc:1918] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.100798 1160 log.cc:1199] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:14.101198 1160 ts_tablet_manager.cc:1939] T 95fce53f9a654fcaa8a52e5733c1b74f P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.102241 1160 tablet_replica.cc:331] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.102366 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 95fce53f9a654fcaa8a52e5733c1b74f (table default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]) successfully deleted
I20250625 07:52:14.102473 1160 raft_consensus.cc:2241] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.102648 1160 raft_consensus.cc:2270] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.103180 1160 ts_tablet_manager.cc:1905] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.111630 1166 ts_tablet_manager.cc:1918] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.111791 1166 log.cc:1199] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:14.112162 1166 ts_tablet_manager.cc:1939] T 55d190e9d11c4589a82fcc7307e9eccb P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.113119 1166 tablet_replica.cc:331] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.113255 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 55d190e9d11c4589a82fcc7307e9eccb (table default.Table_2d6ea477268d47f58b30102e5dc88635 [id=516fdfcb90fd42d690397c77e99b4a38]) successfully deleted
I20250625 07:52:14.113297 1166 raft_consensus.cc:2241] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.113480 1166 raft_consensus.cc:2270] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.116081 1160 ts_tablet_manager.cc:1918] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.116331 1160 log.cc:1199] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/51279fb42ae8480b8511432cb4f808ab
I20250625 07:52:14.116824 1160 ts_tablet_manager.cc:1939] T 51279fb42ae8480b8511432cb4f808ab P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.118789 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b6382963863548398bed90f03b840806
I20250625 07:52:14.121623 1160 tablet_replica.cc:331] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.121821 1160 raft_consensus.cc:2241] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.121982 1160 raft_consensus.cc:2270] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.122928 1160 ts_tablet_manager.cc:1905] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.123929 1167 raft_consensus.cc:491] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.124094 1167 raft_consensus.cc:513] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.124871 1167 leader_election.cc:290] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.125284 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 51279fb42ae8480b8511432cb4f808ab (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:14.125712 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.125615 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de10cbe72c8477ebe83831c869c3df1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:14.126385 851 leader_election.cc:343] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.126451 1166 ts_tablet_manager.cc:1905] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.128742 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1635d57f0e2c477daa163a4fcedfae91" } new_table_name: "default.Table_ac06645d7b71464ab4e4e92402c14a13" modify_external_catalogs: true
I20250625 07:52:14.129137 882 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.133114 882 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.139554 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4c3c180785b64f7d85863c14cfd09ccc
I20250625 07:52:14.140833 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.141615 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9c5249feaf2b436bbf8469b334505134"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.141695 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d4c744fef0e24cb39c4bb0787adb1cbc (DEFAULT_TABLE table=default.Table_b14bab363cb9435f86290aa16ed47a6c [id=29e4ffaed4f148b6b10e38231435c071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.142158 1166 ts_tablet_manager.cc:1918] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:14.142190 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c5249feaf2b436bbf8469b334505134 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.142261 1166 log.cc:1199] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:14.142374 1161 ts_tablet_manager.cc:1939] T 94ef72e8a4f54fd4994dcf4725337478 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:14.142590 1166 ts_tablet_manager.cc:1939] T 1222f2c1027b42c6ad9627359a93d678 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.142697 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.143432 1166 tablet_replica.cc:331] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.143592 1166 raft_consensus.cc:2241] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.143602 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1222f2c1027b42c6ad9627359a93d678 (table default.Table_ca7c2bc86d9f4ad1896150b322f2d1b5 [id=5174ca174ab24d7695eb1fddc9414c74]) successfully deleted
I20250625 07:52:14.143726 1166 raft_consensus.cc:2270] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.144866 1160 ts_tablet_manager.cc:1918] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.144994 1160 log.cc:1199] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e90434606299495081d752df4761f9f8
I20250625 07:52:14.145327 1160 ts_tablet_manager.cc:1939] T e90434606299495081d752df4761f9f8 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.147922 1160 tablet_replica.cc:331] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.148159 1160 raft_consensus.cc:2241] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.148321 1160 raft_consensus.cc:2270] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.148726 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.148882 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de10cbe72c8477ebe83831c869c3df1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:14.149880 1160 ts_tablet_manager.cc:1905] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:14.154183 850 leader_election.cc:343] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.154343 850 leader_election.cc:304] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.154817 1167 raft_consensus.cc:2747] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.154975 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d787caa22cfb4779bb6d87a5b5571157" } new_table_name: "default.Table_f3e586b96cbb4685a99ad12be406febe" modify_external_catalogs: true
I20250625 07:52:14.157706 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c744fef0e24cb39c4bb0787adb1cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.160305 1166 ts_tablet_manager.cc:1905] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:14.161156 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d92962b9e7d14c8d88d775b98e75de35 consensus state has no leader
I20250625 07:52:14.161456 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 9916612cb0234ec3a6df8f22c2b50e7c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.164110 1160 ts_tablet_manager.cc:1918] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.164376 1160 log.cc:1199] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/753404a0a77c46e6a90fbacb6cdfdbe1
I20250625 07:52:14.164544 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c3c180785b64f7d85863c14cfd09ccc from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.164876 1160 ts_tablet_manager.cc:1939] T 753404a0a77c46e6a90fbacb6cdfdbe1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.166285 1160 tablet_replica.cc:331] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.166565 1160 raft_consensus.cc:2241] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.166819 1160 raft_consensus.cc:2270] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.170233 1160 ts_tablet_manager.cc:1905] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.170943 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.171388 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e90434606299495081d752df4761f9f8 (table default.Table_994604bdceea483cb2c00ad1e1c642c5 [id=49b8a037d7f34fe5a6ce7c9fa6a6f258]) successfully deleted
I20250625 07:52:14.178081 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 120) took 1258 ms. Trace:
I20250625 07:52:14.179371 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 31942f0998bb4118a91f476325851d5c (DEFAULT_TABLE table=default.Table_a41e6c6e1d6342a498f4f79c95c7ee52 [id=7686e223b6fb47e79ecb2257550a64b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.179795 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31942f0998bb4118a91f476325851d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.182650 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.183135 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.183550 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08bb47266e8b4a0fa5ec613560b8549e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.187284 1166 ts_tablet_manager.cc:1918] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.187494 1166 log.cc:1199] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:14.187870 1166 ts_tablet_manager.cc:1939] T 572b063d66434e4796f6b86f8221e3c9 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.192780 1163 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.192946 1163 ts_tablet_manager.cc:1397] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.268s user 0.016s sys 0.045s
I20250625 07:52:14.193753 1163 raft_consensus.cc:357] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.193925 1163 raft_consensus.cc:383] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.194001 1163 raft_consensus.cc:738] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.194200 1163 consensus_queue.cc:260] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.194633 1163 ts_tablet_manager.cc:1428] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:14.195593 1163 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.196345 1160 ts_tablet_manager.cc:1918] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.197067 1160 log.cc:1199] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:14.197636 1160 ts_tablet_manager.cc:1939] T 5de10cbe72c8477ebe83831c869c3df1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.198577 1163 tablet_bootstrap.cc:654] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.196923 1161 rpcz_store.cc:276] 0625 07:52:12.895724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:12.895766 (+ 42us) service_pool.cc:225] Handling call
0625 07:52:13.542048 (+646282us) ts_tablet_manager.cc:1145] Deleting tablet 94ef72e8a4f54fd4994dcf4725337478
0625 07:52:13.542055 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:13.698201 (+156146us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x6160000d0954. stack: 00007f96f846d336 00007f96f846d0f0 00007f96f74633ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9703ecd000 00007f97051df093 00007f9707cbc137 00007f9707cee1c4 00007f9707ccfc20 00007f9707ccf9ad 0000000000a7573c 00007f96f84b29df 00007f96f84b5cab 00007f96f84b5b28
0625 07:52:13.999190 (+300989us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.047248 (+ 48058us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.153829 (+106581us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340636544,"tablet-delete.queue_time_us":634750}
I20250625 07:52:14.200659 1160 tablet_replica.cc:331] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.200917 1160 raft_consensus.cc:2241] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.201110 1160 raft_consensus.cc:2270] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.200484 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.202510 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 115) took 1306 ms. Trace:
I20250625 07:52:14.205237 1160 ts_tablet_manager.cc:1905] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.205578 1163 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.205744 1163 ts_tablet_manager.cc:1397] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250625 07:52:14.206908 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.208060 1163 raft_consensus.cc:357] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.208267 1163 raft_consensus.cc:383] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.208338 1163 raft_consensus.cc:738] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.208570 1163 consensus_queue.cc:260] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.209798 1163 ts_tablet_manager.cc:1428] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:14.210109 1163 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.213248 1163 tablet_bootstrap.cc:654] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.214953 1161 rpcz_store.cc:276] 0625 07:52:12.894348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:12.901337 (+ 6989us) service_pool.cc:225] Handling call
0625 07:52:14.200705 (+1299368us) ts_tablet_manager.cc:1145] Deleting tablet 1eab049ad7fc4b5581e5c31a14491306
0625 07:52:14.200717 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.200823 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288095}
I20250625 07:52:14.217439 1161 tablet_replica.cc:331] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:14.217619 1161 raft_consensus.cc:2241] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.218457 1161 raft_consensus.cc:2270] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.218964 1161 ts_tablet_manager.cc:1905] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.219671 1163 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.219815 1163 ts_tablet_manager.cc:1397] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250625 07:52:14.220570 1163 raft_consensus.cc:357] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.222517 1163 raft_consensus.cc:383] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.222615 1163 raft_consensus.cc:738] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.222849 1163 consensus_queue.cc:260] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.223321 1163 ts_tablet_manager.cc:1428] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:14.224925 1163 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.226406 1160 ts_tablet_manager.cc:1918] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.226552 1160 log.cc:1199] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:14.226774 1163 tablet_bootstrap.cc:654] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.229573 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 445bfed532004108a12c8fd78370c1cb (DEFAULT_TABLE table=default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:14.177307 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ad78944352e4274acd8d9889b1d32b8 consensus state has no leader
W20250625 07:52:14.233151 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 61749us
W20250625 07:52:14.233410 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ebb943df93284913b6a520e099399322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebb943df93284913b6a520e099399322 already in progress: creating tablet
I20250625 07:52:14.233469 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dfc00fb110744489ac79c5a1e56bf3c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.233664 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 81976db0a2f94128a89842a12f067231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81976db0a2f94128a89842a12f067231 already in progress: creating tablet
W20250625 07:52:14.233878 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0024618293d74f559b04258f66616044 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0024618293d74f559b04258f66616044 already in progress: creating tablet
W20250625 07:52:14.234079 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7d9d66546d7c42949dfe5db77f1057fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9d66546d7c42949dfe5db77f1057fe already in progress: creating tablet
W20250625 07:52:14.234180 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfc00fb110744489ac79c5a1e56bf3c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.234277 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6b3488a9fb19452dac3241170dd2fee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b3488a9fb19452dac3241170dd2fee2 already in progress: creating tablet
W20250625 07:52:14.234488 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet aec412a15b964060a24631e7ebcf1ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aec412a15b964060a24631e7ebcf1ef7 already in progress: creating tablet
I20250625 07:52:14.234652 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 753404a0a77c46e6a90fbacb6cdfdbe1 (table default.Table_09d0d6f26dac4df085712fb32e0da7c5 [id=4fa6b219e82a4c46bace350861b1cad4]) successfully deleted
I20250625 07:52:14.234839 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5de10cbe72c8477ebe83831c869c3df1 (table default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]) successfully deleted
W20250625 07:52:14.235024 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0452e73865964f709310fd22ac39a5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0452e73865964f709310fd22ac39a5f0 already in progress: creating tablet
I20250625 07:52:14.189250 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 572b063d66434e4796f6b86f8221e3c9 (table default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]) successfully deleted
W20250625 07:52:14.235229 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 21658216da3544a2b8b8f39667782ffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21658216da3544a2b8b8f39667782ffb already in progress: creating tablet
I20250625 07:52:14.200203 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e3f2c828709a47329e6025da1ceb39a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.235358 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 46139.9us
W20250625 07:52:14.235685 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3f2c828709a47329e6025da1ceb39a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.235935 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 09b3830bd506443ab6f06ddd3c6b9974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09b3830bd506443ab6f06ddd3c6b9974 already in progress: creating tablet
W20250625 07:52:14.236172 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 21658216da3544a2b8b8f39667782ffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21658216da3544a2b8b8f39667782ffb already in progress: creating tablet
W20250625 07:52:14.236387 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0452e73865964f709310fd22ac39a5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0452e73865964f709310fd22ac39a5f0 already in progress: creating tablet
W20250625 07:52:14.236577 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b6382963863548398bed90f03b840806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6382963863548398bed90f03b840806 already in progress: creating tablet
W20250625 07:52:14.236799 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4c3c180785b64f7d85863c14cfd09ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c3c180785b64f7d85863c14cfd09ccc already in progress: creating tablet
I20250625 07:52:14.237498 1163 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.238018 1163 ts_tablet_manager.cc:1397] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20250625 07:52:14.200287 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_952886aa3dd24481ac14888ebe7a441d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.238680 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_952886aa3dd24481ac14888ebe7a441d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.238734 1163 raft_consensus.cc:357] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.238920 1163 raft_consensus.cc:383] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.238978 1163 raft_consensus.cc:738] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.239194 1163 consensus_queue.cc:260] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.241523 1163 ts_tablet_manager.cc:1428] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:14.242199 1163 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.242166 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445bfed532004108a12c8fd78370c1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.243830 1163 tablet_bootstrap.cc:654] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.246176 593 master_service.cc:948] ReplaceTablet: received request to replace tablet c7a2b94af59648239cfd1e5ce5e8b968 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.249528 1163 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.249672 1163 ts_tablet_manager.cc:1397] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250625 07:52:14.251201 1163 raft_consensus.cc:357] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.251392 1163 raft_consensus.cc:383] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.251468 1163 raft_consensus.cc:738] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.251670 1163 consensus_queue.cc:260] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.252140 1163 ts_tablet_manager.cc:1428] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:14.252460 1163 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.256683 1163 tablet_bootstrap.cc:654] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:14.256705 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95a763abda074d358f176fd7e7be0e5c consensus state has no leader
I20250625 07:52:14.257148 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_402d22d187b34d4caa19848d9bcc1612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.257504 592 master_service.cc:948] ReplaceTablet: received request to replace tablet b377c8111c9f4ac2ba05e65694d611b9 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:14.257839 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_402d22d187b34d4caa19848d9bcc1612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.257899 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce consensus state has no leader
I20250625 07:52:14.258440 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40651d9551314c82aab23007dc0c10d4" } modify_external_catalogs: true
I20250625 07:52:14.258602 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40651d9551314c82aab23007dc0c10d4" } modify_external_catalogs: true
W20250625 07:52:14.259146 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edda0c3f60f64ed0b965eca52c5b3591 consensus state has no leader
I20250625 07:52:14.259423 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_12188272bb8e4c26bb94db8721d37c26" } modify_external_catalogs: true
I20250625 07:52:14.259618 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_12188272bb8e4c26bb94db8721d37c26" } modify_external_catalogs: true
I20250625 07:52:14.262650 1163 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.262796 1163 ts_tablet_manager.cc:1397] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250625 07:52:14.263478 1163 raft_consensus.cc:357] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.263664 1163 raft_consensus.cc:383] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.263723 1163 raft_consensus.cc:738] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.263952 1163 consensus_queue.cc:260] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.264443 1163 ts_tablet_manager.cc:1428] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:14.264770 1163 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.266477 1163 tablet_bootstrap.cc:654] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.269066 1161 ts_tablet_manager.cc:1918] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.269229 1161 log.cc:1199] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9861eea50d944b2c92be67c3c7c34582
I20250625 07:52:14.269637 1161 ts_tablet_manager.cc:1939] T 9861eea50d944b2c92be67c3c7c34582 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:14.269835 745 tablet_service.cc:1468] Processing CreateTablet for tablet a0518fbc7f52454296ae79a3c92ef800 (DEFAULT_TABLE table=default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.269969 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 121) took 1297 ms. Trace:
I20250625 07:52:14.270074 1161 rpcz_store.cc:276] 0625 07:52:12.972215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:12.973063 (+ 848us) service_pool.cc:225] Handling call
0625 07:52:14.216104 (+1243041us) ts_tablet_manager.cc:1145] Deleting tablet 9861eea50d944b2c92be67c3c7c34582
0625 07:52:14.216115 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.244784 (+ 28669us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.268897 (+ 24113us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.269931 (+ 1034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232276}
I20250625 07:52:14.270293 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0518fbc7f52454296ae79a3c92ef800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.273592 1163 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.273720 1163 ts_tablet_manager.cc:1397] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250625 07:52:14.274482 1163 raft_consensus.cc:357] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.274638 1163 raft_consensus.cc:383] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.274705 1163 raft_consensus.cc:738] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.274906 1163 consensus_queue.cc:260] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.275388 1163 ts_tablet_manager.cc:1428] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:14.275656 1163 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.277426 1163 tablet_bootstrap.cc:654] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.289669 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:14.290056 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b377c8111c9f4ac2ba05e65694d611b9
I20250625 07:52:14.290165 593 catalog_manager.cc:6652] ReplaceTablet: tablet c7a2b94af59648239cfd1e5ce5e8b968 deleted and replaced by tablet 1228018baf3340d99b3f94b150d0d533
I20250625 07:52:14.290400 592 catalog_manager.cc:6652] ReplaceTablet: tablet b377c8111c9f4ac2ba05e65694d611b9 deleted and replaced by tablet 89c703dc3249408faeb522cef0248111
I20250625 07:52:14.290668 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:14.291267 1163 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.291443 1163 ts_tablet_manager.cc:1397] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20250625 07:52:14.292178 1163 raft_consensus.cc:357] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.292380 1163 raft_consensus.cc:383] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.292439 1163 raft_consensus.cc:738] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.292660 1163 consensus_queue.cc:260] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.293167 1163 ts_tablet_manager.cc:1428] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.293223 1160 ts_tablet_manager.cc:1939] T b09dc0b8a496402b82f98daed9af0d12 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.293480 1163 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:14.293613 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 58414.6us
I20250625 07:52:14.293792 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:14.293865 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 09b3830bd506443ab6f06ddd3c6b9974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09b3830bd506443ab6f06ddd3c6b9974 already in progress: creating tablet
W20250625 07:52:14.294097 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6b1eaf0266c04873b185c1c0c2da201c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b1eaf0266c04873b185c1c0c2da201c already in progress: creating tablet
W20250625 07:52:14.294320 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b6382963863548398bed90f03b840806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6382963863548398bed90f03b840806 already in progress: creating tablet
I20250625 07:52:14.294404 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.294943 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.295219 1163 tablet_bootstrap.cc:654] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.295573 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.296159 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.296640 880 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.296959 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.297186 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.297318 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 94ef72e8a4f54fd4994dcf4725337478 (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
W20250625 07:52:14.297533 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1eab049ad7fc4b5581e5c31a14491306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1eab049ad7fc4b5581e5c31a14491306 already in progress: creating tablet
I20250625 07:52:14.297926 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9861eea50d944b2c92be67c3c7c34582 (table default.Table_dfdde9ef755044f7b3d9ab934ca4b231 [id=cc9a674f99634eae89fb70aad7f6ef99]) successfully deleted
I20250625 07:52:14.297922 882 tablet_service.cc:1468] Processing CreateTablet for tablet a0518fbc7f52454296ae79a3c92ef800 (DEFAULT_TABLE table=default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.298592 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.299028 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0518fbc7f52454296ae79a3c92ef800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.301016 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:14.302794 1163 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.302973 1163 ts_tablet_manager.cc:1397] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250625 07:52:14.303851 1163 raft_consensus.cc:357] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.304024 1163 raft_consensus.cc:383] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.304121 1163 raft_consensus.cc:738] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.304343 1163 consensus_queue.cc:260] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.304867 1163 ts_tablet_manager.cc:1428] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.305159 1163 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:14.307862 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c7a2b94af59648239cfd1e5ce5e8b968 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a2b94af59648239cfd1e5ce5e8b968 already in progress: creating tablet
W20250625 07:52:14.308182 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b377c8111c9f4ac2ba05e65694d611b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b377c8111c9f4ac2ba05e65694d611b9 already in progress: creating tablet
W20250625 07:52:14.308562 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c65f257ae653488a8b49b0b523d2cf82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65f257ae653488a8b49b0b523d2cf82 already in progress: creating tablet
I20250625 07:52:14.309285 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.311106 1158 tablet_bootstrap.cc:492] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.311276 1158 ts_tablet_manager.cc:1397] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.563s user 0.056s sys 0.124s
I20250625 07:52:14.311419 1162 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.311579 1162 ts_tablet_manager.cc:1397] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.299s user 0.016s sys 0.028s
I20250625 07:52:14.312026 1158 raft_consensus.cc:357] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.312213 1158 raft_consensus.cc:383] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.312276 1158 raft_consensus.cc:738] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.312533 1158 consensus_queue.cc:260] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.313833 1158 ts_tablet_manager.cc:1428] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.314121 1158 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.315819 1158 tablet_bootstrap.cc:654] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.317422 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a0518fbc7f52454296ae79a3c92ef800 (DEFAULT_TABLE table=default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.317827 1163 tablet_bootstrap.cc:654] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.317836 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0518fbc7f52454296ae79a3c92ef800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.319218 1248 raft_consensus.cc:491] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.319381 1248 raft_consensus.cc:513] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.320034 1248 leader_election.cc:290] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:14.321197 986 leader_election.cc:343] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:14.321381 1162 raft_consensus.cc:357] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.321566 1162 raft_consensus.cc:383] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.321677 1162 raft_consensus.cc:738] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:14.322046 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe already in progress: creating tablet
I20250625 07:52:14.323911 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_808e1666e3d347168df6c5a4301fc6d8" } modify_external_catalogs: true
I20250625 07:52:14.324164 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_808e1666e3d347168df6c5a4301fc6d8" } modify_external_catalogs: true
I20250625 07:52:14.325016 595 master_service.cc:948] ReplaceTablet: received request to replace tablet aec412a15b964060a24631e7ebcf1ef7 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.325119 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0bed2d3626248cfa65f9de3024542fe" } modify_external_catalogs: true
I20250625 07:52:14.325591 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0bed2d3626248cfa65f9de3024542fe" } modify_external_catalogs: true
I20250625 07:52:14.324328 1161 tablet_replica.cc:331] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:14.326334 1161 raft_consensus.cc:2241] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.332476 1161 raft_consensus.cc:2270] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.330628 1158 tablet_bootstrap.cc:492] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.333137 1158 ts_tablet_manager.cc:1397] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20250625 07:52:14.333326 1162 consensus_queue.cc:260] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.333906 1158 raft_consensus.cc:357] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:14.332233 985 leader_election.cc:343] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:14.334589 1162 ts_tablet_manager.cc:1428] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250625 07:52:14.334043 1161 ts_tablet_manager.cc:1905] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.334163 985 leader_election.cc:304] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.334102 1158 raft_consensus.cc:383] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.335412 1158 raft_consensus.cc:738] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.335646 1158 consensus_queue.cc:260] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.336124 1162 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:14.335644 1248 raft_consensus.cc:2747] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.339182 1163 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.339412 1163 ts_tablet_manager.cc:1397] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.008s
I20250625 07:52:14.340523 1163 raft_consensus.cc:357] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.340816 1162 tablet_bootstrap.cc:654] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.340832 1163 raft_consensus.cc:383] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.340930 1163 raft_consensus.cc:738] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.341292 1163 consensus_queue.cc:260] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.349431 1163 ts_tablet_manager.cc:1428] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250625 07:52:14.353394 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:14.354918 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:14.355619 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because the tablet was not found. No further retry: Not found: Tablet not found: 250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:14.357116 1163 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.359534 1163 tablet_bootstrap.cc:654] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.369165 1161 ts_tablet_manager.cc:1918] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.369267 745 tablet_service.cc:1515] Processing DeleteTablet for tablet c44b0ef2ac60457bb56abead72f0182a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.363905 666 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:14.369590 1166 tablet_replica.cc:331] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.369763 1166 raft_consensus.cc:2241] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.365577 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:14.369967 1166 raft_consensus.cc:2270] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.370337 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.370898 1166 ts_tablet_manager.cc:1905] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.372380 745 tablet_service.cc:1515] Processing DeleteTablet for tablet c44b0ef2ac60457bb56abead72f0182a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.373821 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9418b81f010d4f369d3409e9364bc5fa (DEFAULT_TABLE table=default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.374280 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9418b81f010d4f369d3409e9364bc5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.377822 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3c9bae2008e48b199a1d0551ec879c7" } new_table_name: "default.Table_d6bdc3cfe42242a5a341dd866d9ba2ea" modify_external_catalogs: true
I20250625 07:52:14.378283 1166 ts_tablet_manager.cc:1918] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.378403 1166 log.cc:1199] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:14.378775 1166 ts_tablet_manager.cc:1939] T c44b0ef2ac60457bb56abead72f0182a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.379555 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9bbce7454a884554b31faffc79a33b99" } modify_external_catalogs: true
I20250625 07:52:14.379805 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9bbce7454a884554b31faffc79a33b99" } modify_external_catalogs: true
I20250625 07:52:14.380587 1163 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.380627 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_91c2085052354e22a19051706e2493a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.380766 1163 ts_tablet_manager.cc:1397] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
W20250625 07:52:14.381168 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91c2085052354e22a19051706e2493a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.381525 1163 raft_consensus.cc:357] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.381700 1163 raft_consensus.cc:383] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.381778 1163 raft_consensus.cc:738] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.381973 1163 consensus_queue.cc:260] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.382447 1163 ts_tablet_manager.cc:1428] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:14.382798 1163 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.383306 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2e804b4479be4226bc14ca030cdb1144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.383819 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e804b4479be4226bc14ca030cdb1144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.386416 1163 tablet_bootstrap.cc:654] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.388298 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c44b0ef2ac60457bb56abead72f0182a (table default.Table_808e1666e3d347168df6c5a4301fc6d8 [id=f87841f2973841248d450e263c7a5d83]) successfully deleted
W20250625 07:52:14.390681 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because the tablet was not found. No further retry: Not found: Tablet not found: 250175f54ba94a9fbc2455d9cf27fb2e
W20250625 07:52:14.391033 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c44b0ef2ac60457bb56abead72f0182a because the tablet was not found. No further retry: Not found: Tablet not found: c44b0ef2ac60457bb56abead72f0182a
W20250625 07:52:14.392572 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e41e53efb6f342d2b1e4d642f7aaad10 consensus state has no leader
I20250625 07:52:14.394321 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 81976db0a2f94128a89842a12f067231 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.400231 1167 raft_consensus.cc:491] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.400430 1167 raft_consensus.cc:513] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.401101 1167 leader_election.cc:290] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:14.401305 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5c28a064006145eea000e74da10d0794
I20250625 07:52:14.402024 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239e99c6b91d41d68a213a4c8d79470f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:14.402594 851 leader_election.cc:343] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.402617 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.403652 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1351bcbc744a4e77a2f86a998e3073d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.404310 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1351bcbc744a4e77a2f86a998e3073d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.404927 1163 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.405149 1163 ts_tablet_manager.cc:1397] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20250625 07:52:14.405916 1163 raft_consensus.cc:357] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.406193 1163 raft_consensus.cc:383] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.406304 1163 raft_consensus.cc:738] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:14.406584 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5c28a064006145eea000e74da10d0794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c28a064006145eea000e74da10d0794 already in progress: creating tablet
I20250625 07:52:14.406612 1163 consensus_queue.cc:260] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.406970 1162 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.407137 1162 ts_tablet_manager.cc:1397] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.071s user 0.000s sys 0.002s
I20250625 07:52:14.407881 1162 raft_consensus.cc:357] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.408077 1162 raft_consensus.cc:383] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.408170 1162 raft_consensus.cc:738] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.408417 1162 consensus_queue.cc:260] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.408526 1163 ts_tablet_manager.cc:1428] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 07:52:14.409747 1163 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.410007 1162 ts_tablet_manager.cc:1428] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:14.410329 1162 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:14.369285 1158 ts_tablet_manager.cc:1428] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.036s user 0.022s sys 0.010s
I20250625 07:52:14.401902 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239e99c6b91d41d68a213a4c8d79470f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:14.412322 1158 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:14.412966 850 leader_election.cc:343] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.413125 850 leader_election.cc:304] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.413524 1167 raft_consensus.cc:2747] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.413565 1161 log.cc:1199] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9916612cb0234ec3a6df8f22c2b50e7c
I20250625 07:52:14.414039 1161 ts_tablet_manager.cc:1939] T 9916612cb0234ec3a6df8f22c2b50e7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:14.414867 1162 tablet_bootstrap.cc:654] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.415093 1158 tablet_bootstrap.cc:654] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.415234 1163 tablet_bootstrap.cc:654] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.417495 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d31993b4394cfd8832c93ab31b227b with delete_type TABLET_DATA_DELETED (replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.417771 1166 tablet_replica.cc:331] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.417968 1166 raft_consensus.cc:2241] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.418131 1166 raft_consensus.cc:2270] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.418512 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 125) took 1358 ms. Trace:
I20250625 07:52:14.418737 1166 ts_tablet_manager.cc:1905] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.418624 1161 rpcz_store.cc:276] 0625 07:52:13.060065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.060111 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:14.324254 (+1264143us) ts_tablet_manager.cc:1145] Deleting tablet 9916612cb0234ec3a6df8f22c2b50e7c
0625 07:52:14.324266 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.356518 (+ 32252us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.369014 (+ 12496us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.418240 (+ 49226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263033}
I20250625 07:52:14.419114 1161 tablet_replica.cc:331] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:14.419314 1161 raft_consensus.cc:2241] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.419486 1161 raft_consensus.cc:2270] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.420014 1161 ts_tablet_manager.cc:1905] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.425248 1163 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.425395 1163 ts_tablet_manager.cc:1397] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250625 07:52:14.425484 751 tablet_service.cc:1468] Processing CreateTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e (DEFAULT_TABLE table=default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.425868 1166 ts_tablet_manager.cc:1918] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.425941 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250175f54ba94a9fbc2455d9cf27fb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.426025 1166 log.cc:1199] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:14.426456 1166 ts_tablet_manager.cc:1939] T b0d31993b4394cfd8832c93ab31b227b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.426383 1163 raft_consensus.cc:357] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.426568 1163 raft_consensus.cc:383] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.426663 1163 raft_consensus.cc:738] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.426939 1163 consensus_queue.cc:260] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.428030 1162 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.428169 1162 ts_tablet_manager.cc:1397] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20250625 07:52:14.428966 1162 raft_consensus.cc:357] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.429175 1162 raft_consensus.cc:383] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.429282 1162 raft_consensus.cc:738] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.429518 1162 consensus_queue.cc:260] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.430644 1162 ts_tablet_manager.cc:1428] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.430935 1162 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:14.432005 1163 ts_tablet_manager.cc:1428] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:14.432618 1162 tablet_bootstrap.cc:654] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.436385 1163 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.438602 1163 tablet_bootstrap.cc:654] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.438884 750 tablet_service.cc:1468] Processing CreateTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.439344 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a3751333bc48fc8a59b0e7b67a9935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.441387 1163 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.441514 1158 tablet_bootstrap.cc:492] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.441545 1163 ts_tablet_manager.cc:1397] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250625 07:52:14.442271 1163 raft_consensus.cc:357] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.442457 1163 raft_consensus.cc:383] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.442528 1163 raft_consensus.cc:738] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.442785 1163 consensus_queue.cc:260] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.444782 1158 ts_tablet_manager.cc:1397] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.004s
I20250625 07:52:14.445300 1162 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.445470 1162 ts_tablet_manager.cc:1397] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250625 07:52:14.445639 1158 raft_consensus.cc:357] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.446198 1162 raft_consensus.cc:357] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.446405 1162 raft_consensus.cc:383] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.446494 1162 raft_consensus.cc:738] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.446727 1162 consensus_queue.cc:260] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.446892 1163 ts_tablet_manager.cc:1428] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:14.447932 1162 ts_tablet_manager.cc:1428] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.448379 1162 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:14.449991 1163 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.450233 1162 tablet_bootstrap.cc:654] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.451212 1158 raft_consensus.cc:383] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.451342 1158 raft_consensus.cc:738] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.451587 1163 tablet_bootstrap.cc:654] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.451700 1158 consensus_queue.cc:260] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.452785 1158 ts_tablet_manager.cc:1428] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:14.454022 1158 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.455336 747 tablet_service.cc:1468] Processing CreateTablet for tablet 29f2997d1d714100950383e6af04f315 (DEFAULT_TABLE table=default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.455739 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f2997d1d714100950383e6af04f315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.459225 1158 tablet_bootstrap.cc:654] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.462095 1167 raft_consensus.cc:491] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.462333 1167 raft_consensus.cc:513] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.463117 1167 leader_election.cc:290] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:14.464277 851 leader_election.cc:343] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:14.464319 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6fc7c0feb0e04ef28c323323e3007efc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.464797 850 leader_election.cc:343] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.464928 850 leader_election.cc:304] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.465040 750 tablet_service.cc:1468] Processing CreateTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 (DEFAULT_TABLE table=default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.465365 1167 raft_consensus.cc:2747] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.465431 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db9e8d0fefa47bfaeee462dba096b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.467458 1163 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.467515 1161 ts_tablet_manager.cc:1918] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.467617 1163 ts_tablet_manager.cc:1397] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20250625 07:52:14.467657 1161 log.cc:1199] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/be1f3fbffc954fdb8eb5105b5d383d0c
I20250625 07:52:14.468011 1161 ts_tablet_manager.cc:1939] T be1f3fbffc954fdb8eb5105b5d383d0c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:14.468415 1163 raft_consensus.cc:357] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.468605 1163 raft_consensus.cc:383] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.468727 1163 raft_consensus.cc:738] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.468952 1163 consensus_queue.cc:260] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.469523 1163 ts_tablet_manager.cc:1428] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.470417 1158 tablet_bootstrap.cc:492] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.470574 1158 ts_tablet_manager.cc:1397] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250625 07:52:14.470732 747 tablet_service.cc:1468] Processing CreateTablet for tablet cffde34fbba34e44b67985d3480666c3 (DEFAULT_TABLE table=default.Table_4bd7f1d95c4a4f62815742322b59570c [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.471107 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffde34fbba34e44b67985d3480666c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.471400 1158 raft_consensus.cc:357] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.471609 1158 raft_consensus.cc:383] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.471817 1158 raft_consensus.cc:738] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.472075 1158 consensus_queue.cc:260] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.473129 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 131) took 1278 ms. Trace:
I20250625 07:52:14.473259 1161 rpcz_store.cc:276] 0625 07:52:13.194199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.194425 (+ 226us) service_pool.cc:225] Handling call
0625 07:52:14.419021 (+1224596us) ts_tablet_manager.cc:1145] Deleting tablet be1f3fbffc954fdb8eb5105b5d383d0c
0625 07:52:14.419035 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.456708 (+ 37673us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.467381 (+ 10673us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.473085 (+ 5704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224337}
I20250625 07:52:14.473433 1158 ts_tablet_manager.cc:1428] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:14.473013 751 tablet_service.cc:1468] Processing CreateTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 (DEFAULT_TABLE table=default.Table_9c5249feaf2b436bbf8469b334505134 [id=e8fb18d5b2ae454aa4f4a65fab67c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.473737 1158 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.473834 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960ded22e2eb42d9a3494c020a4ea354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.474547 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 132) took 1280 ms. Trace:
I20250625 07:52:14.474686 1161 rpcz_store.cc:276] 0625 07:52:13.194370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.197272 (+ 2902us) service_pool.cc:225] Handling call
0625 07:52:14.473685 (+1276413us) ts_tablet_manager.cc:1145] Deleting tablet e76df9405d91488dad0cfa7f8755bb1c
0625 07:52:14.473693 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.474509 (+ 816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276178}
I20250625 07:52:14.475445 1158 tablet_bootstrap.cc:654] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:14.476446 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_d6bdc3cfe42242a5a341dd866d9ba2ea [id=c2c95c5f1b914c829fa93e8955558fa2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e41e53efb6f342d2b1e4d642f7aaad10 consensus state has no leader
I20250625 07:52:14.476591 1163 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.476928 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 133) took 1279 ms. Trace:
I20250625 07:52:14.478331 1163 tablet_bootstrap.cc:654] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.478714 1161 rpcz_store.cc:276] 0625 07:52:13.197682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.198120 (+ 438us) service_pool.cc:225] Handling call
0625 07:52:14.476803 (+1278683us) ts_tablet_manager.cc:1145] Deleting tablet c6031b9695cd4a99bd0e5391663170ff
0625 07:52:14.476812 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.476898 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278507}
I20250625 07:52:14.479240 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 134) took 1280 ms. Trace:
I20250625 07:52:14.479843 1161 rpcz_store.cc:276] 0625 07:52:13.198502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.200761 (+ 2259us) service_pool.cc:225] Handling call
0625 07:52:14.479093 (+1278332us) ts_tablet_manager.cc:1145] Deleting tablet b09dc0b8a496402b82f98daed9af0d12
0625 07:52:14.479101 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.479197 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278134}
I20250625 07:52:14.481123 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 135) took 1279 ms. Trace:
I20250625 07:52:14.481319 1161 rpcz_store.cc:276] 0625 07:52:13.201157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.201353 (+ 196us) service_pool.cc:225] Handling call
0625 07:52:14.480213 (+1278860us) ts_tablet_manager.cc:1145] Deleting tablet b7139335c7124ab79bc2d0a251984199
0625 07:52:14.480989 (+ 776us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.481089 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278688}
I20250625 07:52:14.485342 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b0d31993b4394cfd8832c93ab31b227b (table default.Table_12188272bb8e4c26bb94db8721d37c26 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:14.485594 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 136) took 1283 ms. Trace:
I20250625 07:52:14.485744 1161 rpcz_store.cc:276] 0625 07:52:13.201729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.201910 (+ 181us) service_pool.cc:225] Handling call
0625 07:52:14.485464 (+1283554us) ts_tablet_manager.cc:1145] Deleting tablet 5f62b03db94d4ee3935866b5dd8bd53f
0625 07:52:14.485475 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.485562 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283380}
I20250625 07:52:14.486204 1161 tablet_replica.cc:331] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:14.486472 1161 raft_consensus.cc:2241] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.486677 1161 raft_consensus.cc:2270] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.487114 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.487670 1166 tablet_replica.cc:331] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.487839 1166 raft_consensus.cc:2241] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.488013 1166 raft_consensus.cc:2270] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.488369 1167 raft_consensus.cc:491] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.488531 1167 raft_consensus.cc:513] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.488656 1166 ts_tablet_manager.cc:1905] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.489223 1167 leader_election.cc:290] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:14.489725 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d31993b4394cfd8832c93ab31b227b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.490170 850 leader_election.cc:343] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.490211 1161 ts_tablet_manager.cc:1905] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:14.490541 851 leader_election.cc:343] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:14.490660 851 leader_election.cc:304] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.491215 1167 raft_consensus.cc:2747] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.495929 1166 ts_tablet_manager.cc:1918] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.496093 1166 log.cc:1199] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:14.496759 1166 ts_tablet_manager.cc:1939] T 5de10cbe72c8477ebe83831c869c3df1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:14.498688 1163 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.499295 1158 tablet_bootstrap.cc:492] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.499486 1158 ts_tablet_manager.cc:1397] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20250625 07:52:14.499526 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 5de10cbe72c8477ebe83831c869c3df1 (table default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]) successfully deleted
I20250625 07:52:14.500308 1158 raft_consensus.cc:357] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.500541 1158 raft_consensus.cc:383] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.500619 1158 raft_consensus.cc:738] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.500946 1158 consensus_queue.cc:260] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.503046 1158 ts_tablet_manager.cc:1428] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:14.503404 1158 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.505148 1158 tablet_bootstrap.cc:654] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.506635 1163 ts_tablet_manager.cc:1397] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20250625 07:52:14.507675 1163 raft_consensus.cc:357] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.507885 1163 raft_consensus.cc:383] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.507964 1163 raft_consensus.cc:738] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.508209 1163 consensus_queue.cc:260] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.508750 1163 ts_tablet_manager.cc:1428] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.509582 1163 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.511276 1163 tablet_bootstrap.cc:654] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.516152 1161 ts_tablet_manager.cc:1918] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.516296 1161 log.cc:1199] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/834d516547b046288e58b7cde4c56e5a
I20250625 07:52:14.516650 1161 ts_tablet_manager.cc:1939] T 834d516547b046288e58b7cde4c56e5a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:14.516923 1163 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.517078 1163 ts_tablet_manager.cc:1397] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20250625 07:52:14.517460 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 138) took 1272 ms. Trace:
I20250625 07:52:14.517560 1161 rpcz_store.cc:276] 0625 07:52:13.244984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.245141 (+ 157us) service_pool.cc:225] Handling call
0625 07:52:14.486128 (+1240987us) ts_tablet_manager.cc:1145] Deleting tablet 834d516547b046288e58b7cde4c56e5a
0625 07:52:14.486139 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.509140 (+ 23001us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.516046 (+ 6906us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.517406 (+ 1360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238531}
I20250625 07:52:14.517834 1163 raft_consensus.cc:357] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.518024 1163 raft_consensus.cc:383] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.518122 1163 raft_consensus.cc:738] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.518363 1163 consensus_queue.cc:260] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.518899 1163 ts_tablet_manager.cc:1428] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:14.519210 1163 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.524250 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.524507 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.525547 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.525676 1160 tablet_replica.cc:331] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.525882 1160 raft_consensus.cc:2241] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.526077 1160 raft_consensus.cc:2270] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.526371 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.526559 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.526626 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.526810 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.527428 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c44b0ef2ac60457bb56abead72f0182a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.522822 1158 tablet_bootstrap.cc:492] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.527755 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c44b0ef2ac60457bb56abead72f0182a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.527802 1158 ts_tablet_manager.cc:1397] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.002s
I20250625 07:52:14.528467 885 tablet_service.cc:1468] Processing CreateTablet for tablet 9418b81f010d4f369d3409e9364bc5fa (DEFAULT_TABLE table=default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.528667 1158 raft_consensus.cc:357] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.528879 1158 raft_consensus.cc:383] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.529001 1158 raft_consensus.cc:738] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.529255 1158 consensus_queue.cc:260] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.529438 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9418b81f010d4f369d3409e9364bc5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.529574 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b17157c58c6a40f7ad4c8dfaa52b410f" } new_table_name: "default.Table_28d38d3401424c46b48d2e93d80f8a81" modify_external_catalogs: true
I20250625 07:52:14.530390 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_989994a848944308835141de4c1d5a6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.530907 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_989994a848944308835141de4c1d5a6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.531229 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1ddc3593237d4cfea5aada771b9228d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.531442 1158 ts_tablet_manager.cc:1428] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20250625 07:52:14.531688 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ddc3593237d4cfea5aada771b9228d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.531726 1158 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.531980 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6960d95c6d2f46529495f6a7be9a0d43" } modify_external_catalogs: true
I20250625 07:52:14.532279 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9936302439324d268bee80fe82d75dd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.532711 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9936302439324d268bee80fe82d75dd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.533833 1158 tablet_bootstrap.cc:654] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.534035 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_586ab82eb678452aac6cef2f1d4c359a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.534475 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_586ab82eb678452aac6cef2f1d4c359a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.534945 1163 tablet_bootstrap.cc:654] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.535380 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.535967 1160 ts_tablet_manager.cc:1905] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.536527 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9418b81f010d4f369d3409e9364bc5fa (DEFAULT_TABLE table=default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.536953 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9418b81f010d4f369d3409e9364bc5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.536913 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fa9514049a2142ca95833a0e0a0bf14b" } modify_external_catalogs: true
I20250625 07:52:14.546577 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9916612cb0234ec3a6df8f22c2b50e7c (table default.Table_9fdc22d5be4b43a4be699388015b9864 [id=7fbebf8268534597bd15d2bc70204cb7]) successfully deleted
I20250625 07:52:14.546542 882 tablet_service.cc:1468] Processing CreateTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e (DEFAULT_TABLE table=default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.546789 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet be1f3fbffc954fdb8eb5105b5d383d0c (table default.Table_5e62955ce44346d290ac1a75c5daa6b6 [id=cabc1d1f237c4617a82c6daa9682e4c3]) successfully deleted
I20250625 07:52:14.546936 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250175f54ba94a9fbc2455d9cf27fb2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.547024 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e76df9405d91488dad0cfa7f8755bb1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e76df9405d91488dad0cfa7f8755bb1c already in progress: creating tablet
W20250625 07:52:14.547214 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c6031b9695cd4a99bd0e5391663170ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6031b9695cd4a99bd0e5391663170ff already in progress: creating tablet
W20250625 07:52:14.547420 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b09dc0b8a496402b82f98daed9af0d12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b09dc0b8a496402b82f98daed9af0d12 already in progress: creating tablet
W20250625 07:52:14.547595 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b7139335c7124ab79bc2d0a251984199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7139335c7124ab79bc2d0a251984199 already in progress: creating tablet
W20250625 07:52:14.547777 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5f62b03db94d4ee3935866b5dd8bd53f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f62b03db94d4ee3935866b5dd8bd53f already in progress: creating tablet
I20250625 07:52:14.547920 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 834d516547b046288e58b7cde4c56e5a (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:14.548281 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b09dc0b8a496402b82f98daed9af0d12 (table default.Table_c931105d39a449fdbdc73dc16830a6a4 [id=7f77bfd2215a491d8fba698477109409]) successfully deleted
W20250625 07:52:14.548452 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4c3c180785b64f7d85863c14cfd09ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c3c180785b64f7d85863c14cfd09ccc already in progress: creating tablet
W20250625 07:52:14.548625 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c7a2b94af59648239cfd1e5ce5e8b968 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a2b94af59648239cfd1e5ce5e8b968 already in progress: creating tablet
I20250625 07:52:14.548820 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.549140 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.553154 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fa9514049a2142ca95833a0e0a0bf14b" } modify_external_catalogs: true
I20250625 07:52:14.557796 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6960d95c6d2f46529495f6a7be9a0d43" } modify_external_catalogs: true
W20250625 07:52:14.559787 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b377c8111c9f4ac2ba05e65694d611b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b377c8111c9f4ac2ba05e65694d611b9 already in progress: creating tablet
I20250625 07:52:14.571436 1160 ts_tablet_manager.cc:1918] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.571594 1160 log.cc:1199] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:14.571954 1160 ts_tablet_manager.cc:1939] T 55d190e9d11c4589a82fcc7307e9eccb P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.575032 1160 tablet_replica.cc:331] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:14.575256 1160 raft_consensus.cc:2241] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.575454 1160 raft_consensus.cc:2270] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.577705 1162 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:14.577879 1162 ts_tablet_manager.cc:1397] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.130s user 0.011s sys 0.000s
I20250625 07:52:14.579363 1160 ts_tablet_manager.cc:1905] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.579259 1162 raft_consensus.cc:357] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.579473 1162 raft_consensus.cc:383] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.579577 1162 raft_consensus.cc:738] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.579869 1162 consensus_queue.cc:260] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.583588 1162 ts_tablet_manager.cc:1428] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
W20250625 07:52:14.563455 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c81396ee6ed7400baa30dcc5d71ba45d consensus state has no leader
W20250625 07:52:14.560081 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c65f257ae653488a8b49b0b523d2cf82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65f257ae653488a8b49b0b523d2cf82 already in progress: creating tablet
I20250625 07:52:14.585201 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 55d190e9d11c4589a82fcc7307e9eccb (table default.Table_2d6ea477268d47f58b30102e5dc88635 [id=516fdfcb90fd42d690397c77e99b4a38]) successfully deleted
I20250625 07:52:14.568459 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a4e6179f3fa6451099d6493d6182ba85" } new_table_name: "default.Table_98fc1c8ebdd141a3b0500bab94206f7b" modify_external_catalogs: true
I20250625 07:52:14.585582 1280 raft_consensus.cc:491] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.585736 1280 raft_consensus.cc:513] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.572950 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dac905b911e84c5cad6ab113b3907866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.586519 1280 leader_election.cc:290] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:14.573495 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b99fa339b63429ca65b10e0fadd2fa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.573163 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 445bfed532004108a12c8fd78370c1cb
I20250625 07:52:14.568900 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e1622741e834193aac4e19c1e66a6d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.576568 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb291540701c4a4fbb5f3fa31fc4b684" } modify_external_catalogs: true
I20250625 07:52:14.587808 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb291540701c4a4fbb5f3fa31fc4b684" } modify_external_catalogs: true
I20250625 07:52:14.563009 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_47373b3c8ede489ca51da1fea3723d35" } new_table_name: "default.Table_4f645096238747a1a97bcd673b5438fd" modify_external_catalogs: true
W20250625 07:52:14.588387 985 leader_election.cc:343] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f774b4ac0c774ee7bf33d13a678cd368
W20250625 07:52:14.588582 986 leader_election.cc:343] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: f774b4ac0c774ee7bf33d13a678cd368
W20250625 07:52:14.586992 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe already in progress: creating tablet
I20250625 07:52:14.588757 986 leader_election.cc:304] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.589210 1280 raft_consensus.cc:2747] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.568960 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 651c74bf9a9b47d6ab5a875a5c304783
I20250625 07:52:14.589871 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.575874 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca7c2bc86d9f4ad1896150b322f2d1b5" } new_table_name: "default.Table_3c1cbfecab414abaa74c66f1773f817e" modify_external_catalogs: true
W20250625 07:52:14.591425 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dac905b911e84c5cad6ab113b3907866 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.591662 1163 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.591835 1163 ts_tablet_manager.cc:1397] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.073s user 0.019s sys 0.006s
I20250625 07:52:14.592335 1162 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:14.592545 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 445bfed532004108a12c8fd78370c1cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 445bfed532004108a12c8fd78370c1cb already in progress: creating tablet
I20250625 07:52:14.592756 588 master_service.cc:948] ReplaceTablet: received request to replace tablet c44b0ef2ac60457bb56abead72f0182a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.593134 1163 raft_consensus.cc:357] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.593353 1163 raft_consensus.cc:383] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.593428 1163 raft_consensus.cc:738] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.593645 1163 consensus_queue.cc:260] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:14.597153 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e1622741e834193aac4e19c1e66a6d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.594113 1163 ts_tablet_manager.cc:1428] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:14.597566 1163 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.599189 1163 tablet_bootstrap.cc:654] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:14.599435 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250175f54ba94a9fbc2455d9cf27fb2e already in progress: creating tablet
I20250625 07:52:14.599911 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:14.600744 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a9945f38ef148caac59acaadcb2ef4c consensus state has no leader
I20250625 07:52:14.596529 1160 ts_tablet_manager.cc:1918] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.600875 1160 log.cc:1199] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:14.601059 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_be2a902089674d54ad4be0f8d749236f" } modify_external_catalogs: true
I20250625 07:52:14.601205 1160 ts_tablet_manager.cc:1939] T c44b0ef2ac60457bb56abead72f0182a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:14.602244 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_be2a902089674d54ad4be0f8d749236f" } modify_external_catalogs: true
W20250625 07:52:14.602576 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 651c74bf9a9b47d6ab5a875a5c304783 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 651c74bf9a9b47d6ab5a875a5c304783 already in progress: creating tablet
I20250625 07:52:14.606910 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 144) took 1241 ms. Trace:
I20250625 07:52:14.607072 1161 rpcz_store.cc:276] 0625 07:52:13.365643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.366568 (+ 925us) service_pool.cc:225] Handling call
0625 07:52:14.594928 (+1228360us) ts_tablet_manager.cc:1145] Deleting tablet 23755f70822740e08ca6a5e5b5571458
0625 07:52:14.594941 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.606841 (+ 11900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1227975}
W20250625 07:52:14.610664 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c770b41e0844786ad81c7b3579f9934 consensus state has no leader
I20250625 07:52:14.611770 1161 tablet_replica.cc:331] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:14.615993 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250175f54ba94a9fbc2455d9cf27fb2e already in progress: creating tablet
I20250625 07:52:14.616806 594 master_service.cc:948] ReplaceTablet: received request to replace tablet b7139335c7124ab79bc2d0a251984199 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.617388 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dfc00fb110744489ac79c5a1e56bf3c5" } modify_external_catalogs: true
I20250625 07:52:14.617573 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dfc00fb110744489ac79c5a1e56bf3c5" } modify_external_catalogs: true
I20250625 07:52:14.617766 882 tablet_service.cc:1468] Processing CreateTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 (DEFAULT_TABLE table=default.Table_9c5249feaf2b436bbf8469b334505134 [id=e8fb18d5b2ae454aa4f4a65fab67c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.618198 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960ded22e2eb42d9a3494c020a4ea354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.622545 596 master_service.cc:948] ReplaceTablet: received request to replace tablet e7100ef72ff14c24be92c21c6eb8a9de from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:14.625552 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b99fa339b63429ca65b10e0fadd2fa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.625646 588 master_service.cc:948] ReplaceTablet: received request to replace tablet d92962b9e7d14c8d88d775b98e75de35 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.625679 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c5249feaf2b436bbf8469b334505134" } new_table_name: "default.Table_786d6b63b6904ffe87ce1f3eb9769954" modify_external_catalogs: true
I20250625 07:52:14.627559 885 tablet_service.cc:1468] Processing CreateTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.627939 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a3751333bc48fc8a59b0e7b67a9935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.630054 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:14.630461 596 catalog_manager.cc:6652] ReplaceTablet: tablet e7100ef72ff14c24be92c21c6eb8a9de deleted and replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:14.631369 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2caae2434af040b2a462dfbb757ac7ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.631778 1163 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.631942 1163 ts_tablet_manager.cc:1397] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.006s
I20250625 07:52:14.632019 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5aad7032c870473ab9869bb56e68b932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.632771 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aad7032c870473ab9869bb56e68b932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.632787 1163 raft_consensus.cc:357] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.632951 1163 raft_consensus.cc:383] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.633042 1163 raft_consensus.cc:738] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.633318 1163 consensus_queue.cc:260] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:14.634730 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fdb6a689ce6440ab7037484e2b9cdc5 consensus state has no leader
W20250625 07:52:14.634894 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2caae2434af040b2a462dfbb757ac7ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.635577 1161 raft_consensus.cc:2241] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.635787 1161 raft_consensus.cc:2270] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.616925 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e (DEFAULT_TABLE table=default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.637434 1163 ts_tablet_manager.cc:1428] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:14.621888 1158 tablet_bootstrap.cc:492] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.625190 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 (DEFAULT_TABLE table=default.Table_9c5249feaf2b436bbf8469b334505134 [id=e8fb18d5b2ae454aa4f4a65fab67c467]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.638231 1158 ts_tablet_manager.cc:1397] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.027s
I20250625 07:52:14.626135 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:14.639204 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 960ded22e2eb42d9a3494c020a4ea354 consensus state has no leader
I20250625 07:52:14.639423 1158 raft_consensus.cc:357] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.639621 1158 raft_consensus.cc:383] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.639688 1158 raft_consensus.cc:738] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.639962 1163 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.640122 1158 consensus_queue.cc:260] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.641441 1158 ts_tablet_manager.cc:1428] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:14.641762 1158 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.643621 1163 tablet_bootstrap.cc:654] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.643958 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250175f54ba94a9fbc2455d9cf27fb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.648761 1161 ts_tablet_manager.cc:1905] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.648840 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960ded22e2eb42d9a3494c020a4ea354. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.651412 1167 raft_consensus.cc:491] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.651577 1167 raft_consensus.cc:513] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.652235 1167 leader_election.cc:290] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.652817 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a3751333bc48fc8a59b0e7b67a9935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.652765 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "572b063d66434e4796f6b86f8221e3c9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.653450 850 leader_election.cc:343] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.654425 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 445bfed532004108a12c8fd78370c1cb from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.664567 881 tablet_service.cc:1468] Processing CreateTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 (DEFAULT_TABLE table=default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.665107 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db9e8d0fefa47bfaeee462dba096b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.670135 1158 tablet_bootstrap.cc:654] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.677379 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 (DEFAULT_TABLE table=default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.678202 881 tablet_service.cc:1468] Processing CreateTablet for tablet 29f2997d1d714100950383e6af04f315 (DEFAULT_TABLE table=default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.678578 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:14.678937 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f2997d1d714100950383e6af04f315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.684468 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:14.684937 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d92962b9e7d14c8d88d775b98e75de35
I20250625 07:52:14.685351 588 catalog_manager.cc:6652] ReplaceTablet: tablet d92962b9e7d14c8d88d775b98e75de35 deleted and replaced by tablet 02ec8c9e6340497abe802c7345f42921
I20250625 07:52:14.685704 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7f196473315441c6bf8b5173ed46f833
W20250625 07:52:14.687085 851 leader_election.cc:343] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:14.687251 851 leader_election.cc:304] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.687695 1167 raft_consensus.cc:2747] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.688406 747 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.689767 745 tablet_service.cc:1468] Processing CreateTablet for tablet 89c703dc3249408faeb522cef0248111 (DEFAULT_TABLE table=default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.689776 1015 tablet_service.cc:1468] Processing CreateTablet for tablet cffde34fbba34e44b67985d3480666c3 (DEFAULT_TABLE table=default.Table_4bd7f1d95c4a4f62815742322b59570c [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.690438 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:38477: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111),
I20250625 07:52:14.690579 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250625 07:52:14.690925 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db9e8d0fefa47bfaeee462dba096b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.690989 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.690941 758 tablet_service.cc:1468] Processing CreateTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 (DEFAULT_TABLE table=default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:14.691581 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e7100ef72ff14c24be92c21c6eb8a9de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7100ef72ff14c24be92c21c6eb8a9de already in progress: creating tablet
I20250625 07:52:14.688957 885 tablet_service.cc:1468] Processing CreateTablet for tablet cffde34fbba34e44b67985d3480666c3 (DEFAULT_TABLE table=default.Table_4bd7f1d95c4a4f62815742322b59570c [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.691977 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffde34fbba34e44b67985d3480666c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.692508 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7ad78944352e4274acd8d9889b1d32b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ad78944352e4274acd8d9889b1d32b8 already in progress: creating tablet
I20250625 07:52:14.692749 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cffde34fbba34e44b67985d3480666c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.693210 1162 tablet_bootstrap.cc:654] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.693439 755 tablet_service.cc:1468] Processing CreateTablet for tablet cb23f900faf6421fb6e72f2b5202d0fd (DEFAULT_TABLE table=default.Table_402d22d187b34d4caa19848d9bcc1612 [id=38f8a949076e4a64acdda251b315ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.694186 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.694397 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1228018baf3340d99b3f94b150d0d533 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.694692 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:14.694804 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3ca0144e3c5142ac9fcbd927b7585a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:14.694860 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.694988 757 tablet_service.cc:1468] Processing CreateTablet for tablet d76681f162074b67a6b886887b5a3d61 (DEFAULT_TABLE table=default.Table_91c2085052354e22a19051706e2493a5 [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.695333 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c703dc3249408faeb522cef0248111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.695647 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb23f900faf6421fb6e72f2b5202d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.695636 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d92962b9e7d14c8d88d775b98e75de35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d92962b9e7d14c8d88d775b98e75de35 already in progress: creating tablet
I20250625 07:52:14.695868 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76681f162074b67a6b886887b5a3d61. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.695876 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7f196473315441c6bf8b5173ed46f833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f196473315441c6bf8b5173ed46f833 already in progress: creating tablet
I20250625 07:52:14.696130 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494cacf2dcbb40d3b2c9f05f64809870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.696557 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1228018baf3340d99b3f94b150d0d533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.708739 1167 raft_consensus.cc:491] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.708999 1167 raft_consensus.cc:513] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.709756 1167 leader_election.cc:290] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.710938 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3488a9fb19452dac3241170dd2fee2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.711500 850 leader_election.cc:343] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.712553 1158 tablet_bootstrap.cc:492] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.712721 1158 ts_tablet_manager.cc:1397] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.003s
I20250625 07:52:14.713836 1158 raft_consensus.cc:357] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.714053 1158 raft_consensus.cc:383] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.714152 1158 raft_consensus.cc:738] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.714453 1158 consensus_queue.cc:260] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.715907 1158 ts_tablet_manager.cc:1428] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:14.716732 1158 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.717610 1280 raft_consensus.cc:491] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.717777 1280 raft_consensus.cc:513] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.718400 1280 leader_election.cc:290] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:14.718952 1158 tablet_bootstrap.cc:654] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:14.719799 985 leader_election.cc:343] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 51279fb42ae8480b8511432cb4f808ab
W20250625 07:52:14.720356 986 leader_election.cc:343] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 51279fb42ae8480b8511432cb4f808ab
I20250625 07:52:14.722627 986 leader_election.cc:304] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.723181 1280 raft_consensus.cc:2747] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.732159 750 tablet_service.cc:1468] Processing CreateTablet for tablet e8fe6405c0ea4d6ca0257a5b485251fa (DEFAULT_TABLE table=default.Table_952886aa3dd24481ac14888ebe7a441d [id=5523889320fc4a09b49b5e365a722575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.733196 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe6405c0ea4d6ca0257a5b485251fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.734442 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6239713a586d43c28c101ba2447d579d (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.734838 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6239713a586d43c28c101ba2447d579d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.737145 749 tablet_service.cc:1468] Processing CreateTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 (DEFAULT_TABLE table=default.Table_2caae2434af040b2a462dfbb757ac7ca [id=01cb78211cdc4a9a9b493e31f8c44a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.737829 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450727d8f33e4eb489969b7974cdeeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.738358 743 tablet_service.cc:1468] Processing CreateTablet for tablet c8c015c202544914b74f1e245b496f82 (DEFAULT_TABLE table=default.Table_9936302439324d268bee80fe82d75dd4 [id=4a285d0804ea403bb60aa49bfde4bd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.738552 748 tablet_service.cc:1468] Processing CreateTablet for tablet 7db110bb6cc146a393d62ddabbe1955a (DEFAULT_TABLE table=default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.739171 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8c015c202544914b74f1e245b496f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.739200 753 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.739578 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.745321 1167 raft_consensus.cc:491] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.745586 1167 raft_consensus.cc:513] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.746572 1167 leader_election.cc:290] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.747385 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.748059 850 leader_election.cc:343] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.748191 752 tablet_service.cc:1468] Processing CreateTablet for tablet a630f8804bbf41de86b08cde419643ff (DEFAULT_TABLE table=default.Table_989994a848944308835141de4c1d5a6d [id=cd9d83bd28e5422188207277162018a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.750864 754 tablet_service.cc:1468] Processing CreateTablet for tablet 02ec8c9e6340497abe802c7345f42921 (DEFAULT_TABLE table=default.Table_1c386358ebca4d0c9f49fc20d6278f3e [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.751636 744 tablet_service.cc:1468] Processing CreateTablet for tablet fcc3afc210a540058294121d2b83c66d (DEFAULT_TABLE table=default.Table_1ddc3593237d4cfea5aada771b9228d5 [id=8a01186fa596413fb75529d0d57a8d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.752034 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3bed3f04c60249aaa389c26788974f58 (DEFAULT_TABLE table=default.Table_586ab82eb678452aac6cef2f1d4c359a [id=8e22a511cbc44f8996d35e514dfda631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.752374 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db110bb6cc146a393d62ddabbe1955a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.758131 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ec8c9e6340497abe802c7345f42921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.759440 758 tablet_service.cc:1468] Processing CreateTablet for tablet b1422987deca41a79f8db666d6961643 (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.749956 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:14.760748 777 raft_consensus.cc:2466] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:14.761477 851 leader_election.cc:304] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.762077 1167 raft_consensus.cc:2802] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:14.762274 1167 raft_consensus.cc:491] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:14.762440 1167 raft_consensus.cc:3058] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:14.765631 1167 raft_consensus.cc:513] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.766017 1158 tablet_bootstrap.cc:492] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.765897 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_97d416631de74a18be102771d2b2a6d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.766217 1158 ts_tablet_manager.cc:1397] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.023s
W20250625 07:52:14.766573 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d416631de74a18be102771d2b2a6d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.766968 1158 raft_consensus.cc:357] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.767179 1158 raft_consensus.cc:383] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.767254 1158 raft_consensus.cc:738] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.768085 1167 leader_election.cc:290] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.768986 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:14.769618 756 tablet_service.cc:1468] Processing CreateTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 (DEFAULT_TABLE table=default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.769048 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3488a9fb19452dac3241170dd2fee2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:14.770539 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc3afc210a540058294121d2b83c66d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.770648 851 leader_election.cc:343] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:14.770735 850 leader_election.cc:343] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.770785 851 leader_election.cc:304] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.770711 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.771257 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
W20250625 07:52:14.771322 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.771413 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed3f04c60249aaa389c26788974f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.771461 777 raft_consensus.cc:3058] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:14.771556 1158 consensus_queue.cc:260] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.772617 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366d3e6fcc9d49be8fbd2cd070632861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.773183 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a630f8804bbf41de86b08cde419643ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.773300 747 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c (DEFAULT_TABLE table=default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.773762 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca0144e3c5142ac9fcbd927b7585a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.774892 1158 ts_tablet_manager.cc:1428] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 07:52:14.775007 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b3432d8524e14973a24118302f5f6ab5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.775280 1158 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:14.775578 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3432d8524e14973a24118302f5f6ab5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.776158 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4d1709d26e43407286b879f4c741db6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.776634 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d1709d26e43407286b879f4c741db6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.778841 1158 tablet_bootstrap.cc:654] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.780469 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d5f29fa855904b50bfd2bee19de5c920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.780997 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5f29fa855904b50bfd2bee19de5c920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.785662 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1422987deca41a79f8db666d6961643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.789412 1167 raft_consensus.cc:2747] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.791249 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1e843117005043efadef968ca11c482f" } modify_external_catalogs: true
I20250625 07:52:14.791464 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1e843117005043efadef968ca11c482f" } modify_external_catalogs: true
I20250625 07:52:14.795270 1280 raft_consensus.cc:491] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.795466 1280 raft_consensus.cc:513] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.796196 1280 leader_election.cc:290] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:14.795996 757 tablet_service.cc:1468] Processing CreateTablet for tablet a33b7cab19244ec08d3821b6cc4c072d (DEFAULT_TABLE table=default.Table_5aad7032c870473ab9869bb56e68b932 [id=dff8e8b2f6da4c39b78f536bcc8467e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.797312 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2caae2434af040b2a462dfbb757ac7ca" } new_table_name: "default.Table_5c23014bc5574cc9a64e940e87188050" modify_external_catalogs: true
W20250625 07:52:14.798643 986 leader_election.cc:343] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:14.801400 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56e7b5805a334eb386aa31e60ece4da8" } modify_external_catalogs: true
I20250625 07:52:14.801604 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56e7b5805a334eb386aa31e60ece4da8" } modify_external_catalogs: true
W20250625 07:52:14.807437 985 leader_election.cc:343] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:14.807590 985 leader_election.cc:304] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.807978 1280 raft_consensus.cc:2747] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.810245 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1c386358ebca4d0c9f49fc20d6278f3e" } new_table_name: "default.Table_58cafcacc8aa4f77a618e6e3f111d687" modify_external_catalogs: true
I20250625 07:52:14.813365 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a41e6c6e1d6342a498f4f79c95c7ee52" } new_table_name: "default.Table_a28f5f4927f34e03b2ea4784b65cfe43" modify_external_catalogs: true
I20250625 07:52:14.817132 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3525675ad75a431abb04ceb227487582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:14.817626 1158 tablet_bootstrap.cc:492] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.817795 1158 ts_tablet_manager.cc:1397] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.003s
W20250625 07:52:14.817977 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3525675ad75a431abb04ceb227487582 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.820318 1158 raft_consensus.cc:357] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.820559 1158 raft_consensus.cc:383] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.820976 1158 raft_consensus.cc:738] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.821301 1158 consensus_queue.cc:260] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.822449 1158 ts_tablet_manager.cc:1428] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:14.825286 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea09ec2fd2a34832ab3e312528a95048"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.825876 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea09ec2fd2a34832ab3e312528a95048 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.833935 595 master_service.cc:948] ReplaceTablet: received request to replace tablet a5a3751333bc48fc8a59b0e7b67a9935 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.837329 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0dd2a46a354a435497fe259c42f87353" } new_table_name: "default.Table_5946e2f2580849578b2bda787459faef" modify_external_catalogs: true
I20250625 07:52:14.840107 596 master_service.cc:948] ReplaceTablet: received request to replace tablet a7cb2bcdd07c42e680eb4f155212423f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.843933 1167 raft_consensus.cc:491] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.844209 1167 raft_consensus.cc:513] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.845844 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 95a763abda074d358f176fd7e7be0e5c
I20250625 07:52:14.846593 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_27c3e9984caf46e0ac463d7dc5b7edc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.847200 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.848461 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 450727d8f33e4eb489969b7974cdeeb1 consensus state has no leader
I20250625 07:52:14.848834 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ceee7626eede43fb91ee9f43f0410e02" } new_table_name: "default.Table_d7220ef926924a20949121c91fa08801" modify_external_catalogs: true
I20250625 07:52:14.852303 1158 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.852468 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3338d1c708b4421c8501740da89afec5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.853130 850 leader_election.cc:343] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.854346 1158 tablet_bootstrap.cc:654] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:14.858736 1167 leader_election.cc:290] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:14.859401 1167 raft_consensus.cc:491] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.860896 1167 raft_consensus.cc:513] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.863950 1161 ts_tablet_manager.cc:1918] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.866137 1163 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:14.870820 1163 ts_tablet_manager.cc:1397] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.231s user 0.028s sys 0.024s
I20250625 07:52:14.872174 1163 raft_consensus.cc:357] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.872589 1163 raft_consensus.cc:383] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.872753 1163 raft_consensus.cc:738] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.873188 1163 consensus_queue.cc:260] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.873529 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 802c22a6346442c88abbeb4b9ef20c57
I20250625 07:52:14.873965 1163 ts_tablet_manager.cc:1428] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:14.877374 1163 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:14.880658 1167 leader_election.cc:290] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:14.882155 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49697a10cdac4c8ebb69f10f4c3437ec consensus state has no leader
I20250625 07:52:14.884740 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a7cb2bcdd07c42e680eb4f155212423f
W20250625 07:52:14.885233 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f63fdf2c8464c8fb58f412263d4e5c5 consensus state has no leader
W20250625 07:52:14.885468 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02ec8c9e6340497abe802c7345f42921 consensus state has no leader
I20250625 07:52:14.885722 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_87e58d9e449a4f69bed998c1f045fbec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.886360 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31942f0998bb4118a91f476325851d5c consensus state has no leader
I20250625 07:52:14.886643 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:14.887082 595 catalog_manager.cc:6652] ReplaceTablet: tablet a5a3751333bc48fc8a59b0e7b67a9935 deleted and replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:14.887439 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 9c770b41e0844786ad81c7b3579f9934 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.887617 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2cdfc224b12c489ebb46a9d55a90c823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:14.888777 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cdfc224b12c489ebb46a9d55a90c823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:14.892562 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87e58d9e449a4f69bed998c1f045fbec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:14.893177 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.893464 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3338d1c708b4421c8501740da89afec5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:14.893862 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.893929 596 catalog_manager.cc:6652] ReplaceTablet: tablet a7cb2bcdd07c42e680eb4f155212423f deleted and replaced by tablet cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:14.892635 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b020ff9f04b4d1796af4ff585cc04aa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:14.893867 1166 tablet_replica.cc:331] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:14.894531 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33b7cab19244ec08d3821b6cc4c072d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.895495 851 leader_election.cc:343] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.895800 851 leader_election.cc:304] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:14.896462 851 leader_election.cc:343] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.896998 1167 raft_consensus.cc:2747] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.897401 1166 raft_consensus.cc:2241] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:14.897627 1166 raft_consensus.cc:2270] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:14.901105 1161 log.cc:1199] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/37f7c060fb8648b9aaa766bbaa1b4ba7
I20250625 07:52:14.901923 1166 ts_tablet_manager.cc:1905] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:14.905395 777 raft_consensus.cc:2466] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:14.909128 851 leader_election.cc:304] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.910058 1167 raft_consensus.cc:2802] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:14.913422 1167 raft_consensus.cc:695] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:14.914491 1167 consensus_queue.cc:237] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.920528 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.870306 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b020ff9f04b4d1796af4ff585cc04aa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:14.921627 850 leader_election.cc:343] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:14.921845 850 leader_election.cc:304] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:14.864975 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 29f2997d1d714100950383e6af04f315 (DEFAULT_TABLE table=default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.922971 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f2997d1d714100950383e6af04f315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.926535 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.928191 1158 tablet_bootstrap.cc:492] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:14.928366 1158 ts_tablet_manager.cc:1397] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.000s
I20250625 07:52:14.929256 1158 raft_consensus.cc:357] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.929349 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.929456 1158 raft_consensus.cc:383] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:14.929530 1158 raft_consensus.cc:738] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:14.929572 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1351bcbc744a4e77a2f86a998e3073d2" } modify_external_catalogs: true
I20250625 07:52:14.929750 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1351bcbc744a4e77a2f86a998e3073d2" } modify_external_catalogs: true
I20250625 07:52:14.929803 1308 raft_consensus.cc:2747] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.929774 1158 consensus_queue.cc:260] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:14.930239 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 572b063d66434e4796f6b86f8221e3c9 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:14.930325 1158 ts_tablet_manager.cc:1428] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:14.930668 1158 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:14.933866 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3432d8524e14973a24118302f5f6ab5" } modify_external_catalogs: true
I20250625 07:52:14.934571 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3432d8524e14973a24118302f5f6ab5" } modify_external_catalogs: true
I20250625 07:52:14.935367 1280 raft_consensus.cc:491] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:14.935511 1280 raft_consensus.cc:513] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:14.936048 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e3f2c828709a47329e6025da1ceb39a4" } modify_external_catalogs: true
I20250625 07:52:14.936192 1280 leader_election.cc:290] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:14.936234 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e3f2c828709a47329e6025da1ceb39a4" } modify_external_catalogs: true
I20250625 07:52:14.936820 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 51279fb42ae8480b8511432cb4f808ab from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:14.937229 985 leader_election.cc:343] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:14.946386 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet cb83af1a0e8f4ec681793470488e8946
I20250625 07:52:14.947558 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:14.948117 594 catalog_manager.cc:6652] ReplaceTablet: tablet 9c770b41e0844786ad81c7b3579f9934 deleted and replaced by tablet cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:14.953215 1158 tablet_bootstrap.cc:654] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:14.953516 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 262034us
I20250625 07:52:14.954443 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 95fce53f9a654fcaa8a52e5733c1b74f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:11 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.955454 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 51279fb42ae8480b8511432cb4f808ab with delete_type TABLET_DATA_DELETED (replaced by tablet 239e99c6b91d41d68a213a4c8d79470f) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:14.955628 986 leader_election.cc:343] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:14.955821 986 leader_election.cc:304] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:14.956125 1280 raft_consensus.cc:2747] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:14.956482 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f774b4ac0c774ee7bf33d13a678cd368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.957211 746 tablet_service.cc:1468] Processing CreateTablet for tablet ba0735d576f947d395bee8d1b865f2fd (DEFAULT_TABLE table=default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.957306 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.958834 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.959026 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:14.959576 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.960244 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.961174 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.964653 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c44b0ef2ac60457bb56abead72f0182a (table default.Table_808e1666e3d347168df6c5a4301fc6d8 [id=f87841f2973841248d450e263c7a5d83]) successfully deleted
W20250625 07:52:14.964984 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5c28a064006145eea000e74da10d0794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c28a064006145eea000e74da10d0794 already in progress: creating tablet
I20250625 07:52:14.965366 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0735d576f947d395bee8d1b865f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:14.965794 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 445bfed532004108a12c8fd78370c1cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 445bfed532004108a12c8fd78370c1cb already in progress: creating tablet
W20250625 07:52:14.966135 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 651c74bf9a9b47d6ab5a875a5c304783 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 651c74bf9a9b47d6ab5a875a5c304783 already in progress: creating tablet
I20250625 07:52:14.966789 882 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.967839 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.968438 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de10cbe72c8477ebe83831c869c3df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.969236 882 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c (DEFAULT_TABLE table=default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.969446 1166 ts_tablet_manager.cc:1918] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:14.969951 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca0144e3c5142ac9fcbd927b7585a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.975093 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c (DEFAULT_TABLE table=default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.975543 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca0144e3c5142ac9fcbd927b7585a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.982805 885 tablet_service.cc:1468] Processing CreateTablet for tablet e8fe6405c0ea4d6ca0257a5b485251fa (DEFAULT_TABLE table=default.Table_952886aa3dd24481ac14888ebe7a441d [id=5523889320fc4a09b49b5e365a722575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.983256 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe6405c0ea4d6ca0257a5b485251fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.991714 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e8fe6405c0ea4d6ca0257a5b485251fa (DEFAULT_TABLE table=default.Table_952886aa3dd24481ac14888ebe7a441d [id=5523889320fc4a09b49b5e365a722575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.992281 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fe6405c0ea4d6ca0257a5b485251fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:14.995463 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:14.997213 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:14.998369 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:14.999290 885 tablet_service.cc:1468] Processing CreateTablet for tablet 89c703dc3249408faeb522cef0248111 (DEFAULT_TABLE table=default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:14.999729 1166 log.cc:1199] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/d51a898202154881ab4ad41da1d519a3
I20250625 07:52:14.999904 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c703dc3249408faeb522cef0248111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.000192 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.001049 1166 ts_tablet_manager.cc:1939] T d51a898202154881ab4ad41da1d519a3 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:15.001667 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 89c703dc3249408faeb522cef0248111 (DEFAULT_TABLE table=default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.002106 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c703dc3249408faeb522cef0248111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.002547 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7db110bb6cc146a393d62ddabbe1955a
I20250625 07:52:15.003301 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.003579 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9097898be312417e8b4a456060da3a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.004124 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9097898be312417e8b4a456060da3a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.005669 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.006366 881 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.007481 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.008083 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.009238 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e498d11f2eb04d23b580c78f5e2f3fcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.009876 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e498d11f2eb04d23b580c78f5e2f3fcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.010040 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.010870 881 tablet_service.cc:1468] Processing CreateTablet for tablet 1228018baf3340d99b3f94b150d0d533 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.011354 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1228018baf3340d99b3f94b150d0d533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.011245 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0f50983dec264e34b2ed511cd2925785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.011852 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f50983dec264e34b2ed511cd2925785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.013132 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fc8939aafc524fba85e19aefef0e51bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.013688 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc8939aafc524fba85e19aefef0e51bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.014020 1162 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:15.014187 1162 ts_tablet_manager.cc:1397] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.422s user 0.009s sys 0.001s
I20250625 07:52:15.014986 1162 raft_consensus.cc:357] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.015187 1162 raft_consensus.cc:383] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.015265 1162 raft_consensus.cc:738] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.015815 1308 raft_consensus.cc:491] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.015995 1308 raft_consensus.cc:513] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.017339 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7139335c7124ab79bc2d0a251984199" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:15.017863 850 leader_election.cc:343] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.020157 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1228018baf3340d99b3f94b150d0d533 (DEFAULT_TABLE table=default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.021042 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1fadd60e35da4326b464f13bef4c7f71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.021399 881 tablet_service.cc:1468] Processing CreateTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 (DEFAULT_TABLE table=default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.021797 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fadd60e35da4326b464f13bef4c7f71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.021838 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494cacf2dcbb40d3b2c9f05f64809870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.025122 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet d51a898202154881ab4ad41da1d519a3 (table default.Table_9e0159dbcce04493a7cc2849bde83764 [id=21758be8aff9485b9313da6b33405f8e]) successfully deleted
I20250625 07:52:15.025350 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.027379 1308 leader_election.cc:290] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.028563 1161 ts_tablet_manager.cc:1939] T 37f7c060fb8648b9aaa766bbaa1b4ba7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:15.029464 1308 raft_consensus.cc:491] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.029487 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 (DEFAULT_TABLE table=default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.029682 1308 raft_consensus.cc:513] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.030469 880 tablet_service.cc:1468] Processing CreateTablet for tablet d76681f162074b67a6b886887b5a3d61 (DEFAULT_TABLE table=default.Table_91c2085052354e22a19051706e2493a5 [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.030877 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76681f162074b67a6b886887b5a3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.030907 1308 leader_election.cc:290] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.031029 1308 raft_consensus.cc:491] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.031174 1308 raft_consensus.cc:513] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.032152 1308 leader_election.cc:290] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:15.032369 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 95a763abda074d358f176fd7e7be0e5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95a763abda074d358f176fd7e7be0e5c already in progress: creating tablet
W20250625 07:52:15.033362 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a5a3751333bc48fc8a59b0e7b67a9935 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5a3751333bc48fc8a59b0e7b67a9935 already in progress: creating tablet
I20250625 07:52:15.033866 1162 consensus_queue.cc:260] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.037731 1162 ts_tablet_manager.cc:1428] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250625 07:52:15.041118 1163 tablet_bootstrap.cc:700] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Time spent opening tablet: real 0.162s user 0.000s sys 0.002s
I20250625 07:52:15.041518 1163 tablet_bootstrap.cc:654] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:15.044487 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 802c22a6346442c88abbeb4b9ef20c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 802c22a6346442c88abbeb4b9ef20c57 already in progress: creating tablet
W20250625 07:52:15.044762 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a7cb2bcdd07c42e680eb4f155212423f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7cb2bcdd07c42e680eb4f155212423f already in progress: creating tablet
W20250625 07:52:15.044965 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9c770b41e0844786ad81c7b3579f9934 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c770b41e0844786ad81c7b3579f9934 already in progress: creating tablet
I20250625 07:52:15.044709 1023 tablet_service.cc:1468] Processing CreateTablet for tablet d76681f162074b67a6b886887b5a3d61 (DEFAULT_TABLE table=default.Table_91c2085052354e22a19051706e2493a5 [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.045395 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2db9e8d0fefa47bfaeee462dba096b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db9e8d0fefa47bfaeee462dba096b49 already in progress: creating tablet
W20250625 07:52:15.045660 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7db110bb6cc146a393d62ddabbe1955a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db110bb6cc146a393d62ddabbe1955a already in progress: creating tablet
I20250625 07:52:15.046677 1026 tablet_service.cc:1468] Processing CreateTablet for tablet cb23f900faf6421fb6e72f2b5202d0fd (DEFAULT_TABLE table=default.Table_402d22d187b34d4caa19848d9bcc1612 [id=38f8a949076e4a64acdda251b315ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.047148 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 (DEFAULT_TABLE table=default.Table_2caae2434af040b2a462dfbb757ac7ca [id=01cb78211cdc4a9a9b493e31f8c44a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.048152 893 tablet_service.cc:1468] Processing CreateTablet for tablet a33b7cab19244ec08d3821b6cc4c072d (DEFAULT_TABLE table=default.Table_5aad7032c870473ab9869bb56e68b932 [id=dff8e8b2f6da4c39b78f536bcc8467e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.048347 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a33b7cab19244ec08d3821b6cc4c072d (DEFAULT_TABLE table=default.Table_5aad7032c870473ab9869bb56e68b932 [id=dff8e8b2f6da4c39b78f536bcc8467e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.048714 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33b7cab19244ec08d3821b6cc4c072d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.049345 886 tablet_service.cc:1468] Processing CreateTablet for tablet b1422987deca41a79f8db666d6961643 (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.049770 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1422987deca41a79f8db666d6961643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.050541 757 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 (DEFAULT_TABLE table=default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.050923 758 tablet_service.cc:1468] Processing CreateTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a (DEFAULT_TABLE table=default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.052402 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_586ab82eb678452aac6cef2f1d4c359a" } new_table_name: "default.Table_a6db9072062d4e5895af0868d99ac31b" modify_external_catalogs: true
I20250625 07:52:15.053264 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.055356 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95" } modify_external_catalogs: true
I20250625 07:52:15.055713 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95" } modify_external_catalogs: true
I20250625 07:52:15.058086 880 tablet_service.cc:1468] Processing CreateTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 (DEFAULT_TABLE table=default.Table_2caae2434af040b2a462dfbb757ac7ca [id=01cb78211cdc4a9a9b493e31f8c44a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.058763 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450727d8f33e4eb489969b7974cdeeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.065505 1027 tablet_service.cc:1468] Processing CreateTablet for tablet b1422987deca41a79f8db666d6961643 (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.066941 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 6239713a586d43c28c101ba2447d579d (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.067474 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bed3f04c60249aaa389c26788974f58 consensus state has no leader
I20250625 07:52:15.068096 893 tablet_service.cc:1468] Processing CreateTablet for tablet a630f8804bbf41de86b08cde419643ff (DEFAULT_TABLE table=default.Table_989994a848944308835141de4c1d5a6d [id=cd9d83bd28e5422188207277162018a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.068269 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a630f8804bbf41de86b08cde419643ff (DEFAULT_TABLE table=default.Table_989994a848944308835141de4c1d5a6d [id=cd9d83bd28e5422188207277162018a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.068523 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a630f8804bbf41de86b08cde419643ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.068873 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6239713a586d43c28c101ba2447d579d (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.069273 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6239713a586d43c28c101ba2447d579d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.071924 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:15.080053 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1228018baf3340d99b3f94b150d0d533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.080708 1158 tablet_bootstrap.cc:492] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:15.080862 1158 ts_tablet_manager.cc:1397] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.000s
I20250625 07:52:15.080880 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5117224af8bf44ee866ec4497721cf63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.081499 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5117224af8bf44ee866ec4497721cf63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.082257 1158 raft_consensus.cc:357] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.082446 1158 raft_consensus.cc:383] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.082542 1158 raft_consensus.cc:738] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.082753 1158 consensus_queue.cc:260] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.082962 1280 raft_consensus.cc:491] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.083120 1280 raft_consensus.cc:513] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.083485 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a33b7cab19244ec08d3821b6cc4c072d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.083443 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_38b762c38ec54ae09b10d5524e53a950"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.083846 1280 leader_election.cc:290] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:15.083981 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38b762c38ec54ae09b10d5524e53a950 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.085098 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb23f900faf6421fb6e72f2b5202d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.086787 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.087327 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.088173 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e1622741e834193aac4e19c1e66a6d4" } modify_external_catalogs: true
I20250625 07:52:15.088335 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e1622741e834193aac4e19c1e66a6d4" } modify_external_catalogs: true
W20250625 07:52:15.088521 985 leader_election.cc:343] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:15.089309 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b3830bd506443ab6f06ddd3c6b9974" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.089921 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d51a898202154881ab4ad41da1d519a3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.090570 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a630f8804bbf41de86b08cde419643ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.090785 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d76681f162074b67a6b886887b5a3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.091176 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494cacf2dcbb40d3b2c9f05f64809870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.095412 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_236dd4e9c793494b8736319f9247a3da" } new_table_name: "default.Table_5b4da047314e43ebbb8920e13911845c" modify_external_catalogs: true
I20250625 07:52:15.095966 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 450727d8f33e4eb489969b7974cdeeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.096288 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1fadd60e35da4326b464f13bef4c7f71" } new_table_name: "default.Table_d14c6b22d6b745c9aadedb8f7725f4b4" modify_external_catalogs: true
W20250625 07:52:15.097088 850 leader_election.cc:343] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:15.097329 850 leader_election.cc:343] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.097628 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3b0a3a4ca089491d8571b826333d5d9e" } new_table_name: "default.Table_f562abe21a314a6aaf1bf1574b9b908d" modify_external_catalogs: true
I20250625 07:52:15.098752 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_28d38d3401424c46b48d2e93d80f8a81" } new_table_name: "default.Table_3aec9b6017944d5f8293424556b380d3" modify_external_catalogs: true
I20250625 07:52:15.099917 1280 raft_consensus.cc:491] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.100039 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 145) took 1733 ms. Trace:
I20250625 07:52:15.101521 1280 raft_consensus.cc:513] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.102350 1280 leader_election.cc:290] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:15.103240 893 tablet_service.cc:1468] Processing CreateTablet for tablet 3bed3f04c60249aaa389c26788974f58 (DEFAULT_TABLE table=default.Table_586ab82eb678452aac6cef2f1d4c359a [id=8e22a511cbc44f8996d35e514dfda631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.103653 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed3f04c60249aaa389c26788974f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.109800 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:15.116479 889 tablet_service.cc:1468] Processing CreateTablet for tablet c8c015c202544914b74f1e245b496f82 (DEFAULT_TABLE table=default.Table_9936302439324d268bee80fe82d75dd4 [id=4a285d0804ea403bb60aa49bfde4bd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.116995 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8c015c202544914b74f1e245b496f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.120908 1167 raft_consensus.cc:491] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.121248 1167 raft_consensus.cc:513] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.122102 1167 leader_election.cc:290] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:15.124912 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1afe90251c964e369d05e82c7db19326 has no consensus state
W20250625 07:52:15.125386 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba24bbd0c4694930978a7750f07601d9 consensus state has no leader
W20250625 07:52:15.125718 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 875fa1b46903452eaed1f2235d63dfdf consensus state has no leader
W20250625 07:52:15.128122 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c81396ee6ed7400baa30dcc5d71ba45d consensus state has no leader
I20250625 07:52:15.129408 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1422987deca41a79f8db666d6961643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.130406 882 tablet_service.cc:1468] Processing CreateTablet for tablet cb23f900faf6421fb6e72f2b5202d0fd (DEFAULT_TABLE table=default.Table_402d22d187b34d4caa19848d9bcc1612 [id=38f8a949076e4a64acdda251b315ac87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.130769 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6239713a586d43c28c101ba2447d579d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.131168 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb23f900faf6421fb6e72f2b5202d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.133294 1161 rpcz_store.cc:276] 0625 07:52:13.366394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.367576 (+ 1182us) service_pool.cc:225] Handling call
0625 07:52:14.610231 (+1242655us) ts_tablet_manager.cc:1145] Deleting tablet 37f7c060fb8648b9aaa766bbaa1b4ba7
0625 07:52:14.610243 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:14.669794 (+ 59551us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:14.863743 (+193949us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:15.099920 (+236177us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x61800000d518. stack: 00007f96f846d336 00007f96f846d0f0 00007f96f74633ad 0000000000971004 0000000000970fcd 000000000097099a 00007f96f8455c3d 00007f96f8456cae 00007f970baeb80d 00007f970ba6d0f3 00007f9707cbc5a3 00007f9707cee1c4 00007f9707ccfc20 00007f9707ccf9ad 0000000000a7573c 00007f96f84b29df
0625 07:52:15.099993 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123632384,"tablet-delete.queue_time_us":1227017}
W20250625 07:52:15.133742 985 leader_election.cc:343] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:15.134805 1166 tablet_replica.cc:331] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:15.135021 1166 raft_consensus.cc:2241] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.135037 893 tablet_service.cc:1468] Processing CreateTablet for tablet fcc3afc210a540058294121d2b83c66d (DEFAULT_TABLE table=default.Table_1ddc3593237d4cfea5aada771b9228d5 [id=8a01186fa596413fb75529d0d57a8d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.135427 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc3afc210a540058294121d2b83c66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.135993 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e6509d5448ab4dba96e3a6571d4963a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.136557 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6509d5448ab4dba96e3a6571d4963a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.138924 1166 raft_consensus.cc:2270] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.138974 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cb899cd4017844fa834aafebadd0af7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.139477 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb899cd4017844fa834aafebadd0af7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.139875 884 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.140277 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.150668 1021 tablet_service.cc:1468] Processing CreateTablet for tablet fcc3afc210a540058294121d2b83c66d (DEFAULT_TABLE table=default.Table_1ddc3593237d4cfea5aada771b9228d5 [id=8a01186fa596413fb75529d0d57a8d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.150669 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7db110bb6cc146a393d62ddabbe1955a (DEFAULT_TABLE table=default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.150763 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e28f36e9e034b489bf92ac42b5f0e28" } modify_external_catalogs: true
I20250625 07:52:15.150959 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e28f36e9e034b489bf92ac42b5f0e28" } modify_external_catalogs: true
I20250625 07:52:15.150933 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 3bed3f04c60249aaa389c26788974f58 (DEFAULT_TABLE table=default.Table_586ab82eb678452aac6cef2f1d4c359a [id=8e22a511cbc44f8996d35e514dfda631]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.151105 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db110bb6cc146a393d62ddabbe1955a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.151155 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c8c015c202544914b74f1e245b496f82 (DEFAULT_TABLE table=default.Table_9936302439324d268bee80fe82d75dd4 [id=4a285d0804ea403bb60aa49bfde4bd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.151515 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 7db110bb6cc146a393d62ddabbe1955a (DEFAULT_TABLE table=default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.156517 1166 ts_tablet_manager.cc:1905] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.157032 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2771159858fc41a1ad930229986edd92 (DEFAULT_TABLE table=default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.158365 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.159054 750 tablet_service.cc:1468] Processing CreateTablet for tablet be7b755d95b14ecea7a691003455f381 (DEFAULT_TABLE table=default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.159627 745 tablet_service.cc:1468] Processing CreateTablet for tablet 743123ffcf8240dc82bea7178559b7ea (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.160645 756 tablet_service.cc:1468] Processing CreateTablet for tablet 541c983d64f348539979f74199f79e59 (DEFAULT_TABLE table=default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.161949 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec29e4294864fbabdff5a04dcf5b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.161980 890 tablet_service.cc:1468] Processing CreateTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 (DEFAULT_TABLE table=default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.162325 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366d3e6fcc9d49be8fbd2cd070632861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.162839 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a558e0a5403d4489892dfce9587e090c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.163029 746 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.163225 747 tablet_service.cc:1468] Processing CreateTablet for tablet cbdbd9230210455ba615d51f1a98267b (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.163405 743 tablet_service.cc:1468] Processing CreateTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 (DEFAULT_TABLE table=default.Table_d5f29fa855904b50bfd2bee19de5c920 [id=4ff84ce52a874a5792b79f35f2016f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.163633 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f566cc9cdd447798a5cb408ef9cd35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.164907 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2771159858fc41a1ad930229986edd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.166280 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743123ffcf8240dc82bea7178559b7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.166713 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 152) took 1690 ms. Trace:
I20250625 07:52:15.168874 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7b755d95b14ecea7a691003455f381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.170056 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7139335c7124ab79bc2d0a251984199" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.171833 884 tablet_service.cc:1468] Processing CreateTablet for tablet 02ec8c9e6340497abe802c7345f42921 (DEFAULT_TABLE table=default.Table_1c386358ebca4d0c9f49fc20d6278f3e [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.172117 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc3afc210a540058294121d2b83c66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.172250 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ec8c9e6340497abe802c7345f42921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.173933 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8c015c202544914b74f1e245b496f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.174104 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a0518fbc7f52454296ae79a3c92ef800
I20250625 07:52:15.173617 749 tablet_service.cc:1468] Processing CreateTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.175673 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541c983d64f348539979f74199f79e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.175907 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bed3f04c60249aaa389c26788974f58. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.176568 986 leader_election.cc:343] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:15.176633 1162 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:15.176728 986 leader_election.cc:304] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:15.177175 1280 raft_consensus.cc:2747] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:15.177488 851 leader_election.cc:343] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.177626 851 leader_election.cc:304] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.177892 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:15.178066 851 leader_election.cc:343] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: d51a898202154881ab4ad41da1d519a3
I20250625 07:52:15.178174 851 leader_election.cc:304] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.178395 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:15.178551 851 leader_election.cc:343] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a558e0a5403d4489892dfce9587e090c
I20250625 07:52:15.179021 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.181144 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db110bb6cc146a393d62ddabbe1955a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.182353 986 leader_election.cc:343] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:15.182518 986 leader_election.cc:304] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:15.183072 1280 raft_consensus.cc:2747] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.184269 1167 raft_consensus.cc:2747] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.185484 1167 raft_consensus.cc:2747] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.186961 1167 raft_consensus.cc:491] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.187314 1050 raft_consensus.cc:2466] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:15.187806 1167 raft_consensus.cc:513] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.189246 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 02ec8c9e6340497abe802c7345f42921 (DEFAULT_TABLE table=default.Table_1c386358ebca4d0c9f49fc20d6278f3e [id=eecbc272d0844eaaaa51584b812f2321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.189560 890 tablet_service.cc:1468] Processing CreateTablet for tablet ba0735d576f947d395bee8d1b865f2fd (DEFAULT_TABLE table=default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.189582 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b3830bd506443ab6f06ddd3c6b9974" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.189744 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ec8c9e6340497abe802c7345f42921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.189980 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0735d576f947d395bee8d1b865f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.192742 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a94dc1432d4cbdbf3eecacb1429c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.193531 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:15.194969 851 leader_election.cc:343] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.195123 851 leader_election.cc:304] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.195431 1312 raft_consensus.cc:2747] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.197446 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4d1709d26e43407286b879f4c741db6e" } modify_external_catalogs: true
I20250625 07:52:15.197681 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4d1709d26e43407286b879f4c741db6e" } modify_external_catalogs: true
I20250625 07:52:15.198534 744 tablet_service.cc:1468] Processing CreateTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 (DEFAULT_TABLE table=default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.199239 1163 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.199407 1163 ts_tablet_manager.cc:1397] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.325s user 0.000s sys 0.009s
I20250625 07:52:15.200209 1163 raft_consensus.cc:357] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.200424 1163 raft_consensus.cc:383] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.200488 1163 raft_consensus.cc:738] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.202519 850 leader_election.cc:304] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7
I20250625 07:52:15.202816 1312 raft_consensus.cc:2802] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:15.202927 1312 raft_consensus.cc:491] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:15.203035 1312 raft_consensus.cc:3058] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:15.203944 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 (DEFAULT_TABLE table=default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.204489 1161 rpcz_store.cc:276] 0625 07:52:13.453436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.453631 (+ 195us) service_pool.cc:225] Handling call
0625 07:52:15.133612 (+1679981us) ts_tablet_manager.cc:1145] Deleting tablet 53c6c51a90754878977aeeaacf4773a6
0625 07:52:15.133623 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.143933 (+ 10310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671088}
I20250625 07:52:15.206025 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.206718 1163 consensus_queue.cc:260] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.207147 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4f645096238747a1a97bcd673b5438fd" } modify_external_catalogs: true
I20250625 07:52:15.207365 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4f645096238747a1a97bcd673b5438fd" } modify_external_catalogs: true
I20250625 07:52:15.208034 1163 ts_tablet_manager.cc:1428] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 07:52:15.208521 1163 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.212246 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.213675 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.217753 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.218075 1160 tablet_replica.cc:331] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.218269 1160 raft_consensus.cc:2241] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.218343 1312 raft_consensus.cc:513] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.218454 1160 raft_consensus.cc:2270] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.219002 1312 leader_election.cc:290] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.219049 1160 ts_tablet_manager.cc:1905] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.221607 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e4bf78218f4b82b4f16f59b1704256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.225283 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.225464 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6fc7c0feb0e04ef28c323323e3007efc with delete_type TABLET_DATA_DELETED (replaced by tablet b769ac57346a427abd22214eebc0d324) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.226419 1160 ts_tablet_manager.cc:1918] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.226541 1160 log.cc:1199] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:15.226965 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.227212 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.227437 895 tablet_service.cc:1515] Processing DeleteTablet for tablet a558e0a5403d4489892dfce9587e090c with delete_type TABLET_DATA_DELETED (replaced by tablet 1eab049ad7fc4b5581e5c31a14491306) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.227659 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c44b0ef2ac60457bb56abead72f0182a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.227844 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.227908 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.228070 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.228288 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.228475 895 tablet_service.cc:1515] Processing DeleteTablet for tablet b0d31993b4394cfd8832c93ab31b227b with delete_type TABLET_DATA_DELETED (replaced by tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.228528 888 tablet_service.cc:1468] Processing CreateTablet for tablet 743123ffcf8240dc82bea7178559b7ea (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.228718 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.229032 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743123ffcf8240dc82bea7178559b7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.229069 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6c3be49be14ecbb3c26ef107ede015. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.229959 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdbd9230210455ba615d51f1a98267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.236426 1167 leader_election.cc:290] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.237001 1160 ts_tablet_manager.cc:1939] T 572b063d66434e4796f6b86f8221e3c9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.237962 1160 tablet_replica.cc:331] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.238164 1160 raft_consensus.cc:2241] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.238302 1160 raft_consensus.cc:2270] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.238777 1160 ts_tablet_manager.cc:1905] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.241391 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.241709 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.242184 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.243364 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aacce885498f4090b6087bc73c1ba997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.244046 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aacce885498f4090b6087bc73c1ba997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.244383 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 37f7c060fb8648b9aaa766bbaa1b4ba7 (table default.Table_b7677386183a479f9714604d7c2ab058 [id=e4b3ee29eafd4621811d8db11bdd65c4]) successfully deleted
I20250625 07:52:15.246043 894 tablet_service.cc:1468] Processing CreateTablet for tablet 541c983d64f348539979f74199f79e59 (DEFAULT_TABLE table=default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.246114 888 tablet_service.cc:1468] Processing CreateTablet for tablet 2771159858fc41a1ad930229986edd92 (DEFAULT_TABLE table=default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.245909 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a (DEFAULT_TABLE table=default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.246526 887 tablet_service.cc:1468] Processing CreateTablet for tablet cbdbd9230210455ba615d51f1a98267b (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.246730 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541c983d64f348539979f74199f79e59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.246846 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.246740 895 tablet_service.cc:1468] Processing CreateTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.247717 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f566cc9cdd447798a5cb408ef9cd35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.248081 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a94dc1432d4cbdbf3eecacb1429c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.249114 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdbd9230210455ba615d51f1a98267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.249563 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_210190c86c3c4587986ad321a3d67e0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.250255 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_210190c86c3c4587986ad321a3d67e0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.252002 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_224bc28d99ba48a09163db6d3106dfc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.252548 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_224bc28d99ba48a09163db6d3106dfc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.253023 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2771159858fc41a1ad930229986edd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.254607 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_000e46983c3d4c018a661c9efae36202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.255082 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2a4d9d86ac944371a48bcdd835f6c025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.255618 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a4d9d86ac944371a48bcdd835f6c025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:15.256986 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_000e46983c3d4c018a661c9efae36202 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.264765 1311 raft_consensus.cc:491] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.264981 1311 raft_consensus.cc:513] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.267089 1311 leader_election.cc:290] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.267671 885 tablet_service.cc:1468] Processing CreateTablet for tablet be7b755d95b14ecea7a691003455f381 (DEFAULT_TABLE table=default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.270459 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (DEFAULT_TABLE table=default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.276909 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 366d3e6fcc9d49be8fbd2cd070632861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.277789 880 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 (DEFAULT_TABLE table=default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.278213 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.280764 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b09dc0b8a496402b82f98daed9af0d12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.281307 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 (DEFAULT_TABLE table=default.Table_d5f29fa855904b50bfd2bee19de5c920 [id=4ff84ce52a874a5792b79f35f2016f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.281742 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.282308 1012 tablet_service.cc:1468] Processing CreateTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 (DEFAULT_TABLE table=default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.284145 1158 ts_tablet_manager.cc:1428] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.123s user 0.011s sys 0.018s
I20250625 07:52:15.284632 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 153) took 1815 ms. Trace:
I20250625 07:52:15.284839 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.285063 889 tablet_service.cc:1468] Processing CreateTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 (DEFAULT_TABLE table=default.Table_d5f29fa855904b50bfd2bee19de5c920 [id=4ff84ce52a874a5792b79f35f2016f1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.285319 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.285379 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7b755d95b14ecea7a691003455f381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.285653 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e4bf78218f4b82b4f16f59b1704256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.286602 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec29e4294864fbabdff5a04dcf5b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.287050 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1222f2c1027b42c6ad9627359a93d678" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.287542 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.287505 1011 tablet_service.cc:1468] Processing CreateTablet for tablet ba0735d576f947d395bee8d1b865f2fd (DEFAULT_TABLE table=default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.287714 1315 raft_consensus.cc:491] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.287786 1161 rpcz_store.cc:276] 0625 07:52:13.455868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.455968 (+ 100us) service_pool.cc:225] Handling call
0625 07:52:15.228331 (+1772363us) ts_tablet_manager.cc:1145] Deleting tablet 6606a9559e6b47d28669083af5aee5bd
0625 07:52:15.228348 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.271026 (+ 42678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742471}
I20250625 07:52:15.287883 1315 raft_consensus.cc:513] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.288115 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.288255 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23755f70822740e08ca6a5e5b5571458" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.288734 1158 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:15.289453 1315 leader_election.cc:290] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:15.292263 892 tablet_service.cc:1468] Processing CreateTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 (DEFAULT_TABLE table=default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.292654 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6c3be49be14ecbb3c26ef107ede015. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.292827 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a558e0a5403d4489892dfce9587e090c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:15.293962 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:15.300072 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.304204 1311 raft_consensus.cc:491] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.304378 1311 raft_consensus.cc:513] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.305176 1311 leader_election.cc:290] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.305550 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.305791 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.307403 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86ea9df068a94d9187f2f9620e371a39" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:15.307981 850 leader_election.cc:343] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:15.308197 850 leader_election.cc:343] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:15.308355 850 leader_election.cc:343] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.310156 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.315403 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 743123ffcf8240dc82bea7178559b7ea (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.316800 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d5b724ca63c436ba1ee776b8df3cec9" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.317060 1050 raft_consensus.cc:3058] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:15.318074 778 raft_consensus.cc:2391] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f2cdaf87b9d4b2a919098bf65efd513 in current term 1: Already voted for candidate 429026e8d98d46339db143f84897a092 in this term.
W20250625 07:52:15.321633 851 leader_election.cc:343] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a558e0a5403d4489892dfce9587e090c
I20250625 07:52:15.321702 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a (DEFAULT_TABLE table=default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.321892 851 leader_election.cc:343] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:15.322000 851 leader_election.cc:304] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.324270 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86ea9df068a94d9187f2f9620e371a39" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:15.324792 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 80103.9us
W20250625 07:52:15.325310 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 53c6c51a90754878977aeeaacf4773a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53c6c51a90754878977aeeaacf4773a6 already in progress: creating tablet
W20250625 07:52:15.325363 851 leader_election.cc:343] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.325474 851 leader_election.cc:304] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:15.325551 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6606a9559e6b47d28669083af5aee5bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6606a9559e6b47d28669083af5aee5bd already in progress: creating tablet
I20250625 07:52:15.325641 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 (DEFAULT_TABLE table=default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.325749 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e7100ef72ff14c24be92c21c6eb8a9de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7100ef72ff14c24be92c21c6eb8a9de already in progress: creating tablet
W20250625 07:52:15.326081 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3ca0144e3c5142ac9fcbd927b7585a3c because the tablet was not found. No further retry: Not found: Tablet not found: 3ca0144e3c5142ac9fcbd927b7585a3c
W20250625 07:52:15.326313 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7ad78944352e4274acd8d9889b1d32b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ad78944352e4274acd8d9889b1d32b8 already in progress: creating tablet
I20250625 07:52:15.326380 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23755f70822740e08ca6a5e5b5571458" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:15.326483 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d92962b9e7d14c8d88d775b98e75de35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d92962b9e7d14c8d88d775b98e75de35 already in progress: creating tablet
I20250625 07:52:15.326597 986 leader_election.cc:304] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:15.326668 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7f196473315441c6bf8b5173ed46f833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f196473315441c6bf8b5173ed46f833 already in progress: creating tablet
W20250625 07:52:15.326952 851 leader_election.cc:343] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.327068 851 leader_election.cc:304] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.327342 1312 raft_consensus.cc:2747] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.327522 1312 raft_consensus.cc:2747] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.327668 1312 raft_consensus.cc:2747] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.327788 1312 raft_consensus.cc:491] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:15.327771 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 802c22a6346442c88abbeb4b9ef20c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 802c22a6346442c88abbeb4b9ef20c57 already in progress: creating tablet
I20250625 07:52:15.327919 1312 raft_consensus.cc:513] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:15.328069 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a7cb2bcdd07c42e680eb4f155212423f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7cb2bcdd07c42e680eb4f155212423f already in progress: creating tablet
W20250625 07:52:15.328284 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 95a763abda074d358f176fd7e7be0e5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95a763abda074d358f176fd7e7be0e5c already in progress: creating tablet
I20250625 07:52:15.328558 1312 leader_election.cc:290] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.329020 1163 tablet_bootstrap.cc:700] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Time spent opening tablet: real 0.118s user 0.001s sys 0.001s
I20250625 07:52:15.329066 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e4bf78218f4b82b4f16f59b1704256. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.329402 1163 tablet_bootstrap.cc:654] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.333648 1162 tablet_bootstrap.cc:654] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.335218 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9a94dc1432d4cbdbf3eecacb1429c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.341251 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012df7beedb1446ca5e533d6624afc10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.341557 1160 ts_tablet_manager.cc:1918] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.341702 1160 log.cc:1199] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2b020ff9f04b4d1796af4ff585cc04aa
I20250625 07:52:15.342159 1160 ts_tablet_manager.cc:1939] T 2b020ff9f04b4d1796af4ff585cc04aa P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:15.342664 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9c770b41e0844786ad81c7b3579f9934 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c770b41e0844786ad81c7b3579f9934 already in progress: creating tablet
I20250625 07:52:15.342864 1315 raft_consensus.cc:3058] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:15.343243 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 572b063d66434e4796f6b86f8221e3c9 (table default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]) successfully deleted
I20250625 07:52:15.343513 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.342283 1167 raft_consensus.cc:491] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.346249 1167 raft_consensus.cc:513] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.346917 1167 leader_election.cc:290] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:15.344290 1163 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.347167 1163 ts_tablet_manager.cc:1397] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.002s
I20250625 07:52:15.347570 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012df7beedb1446ca5e533d6624afc10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.348500 1163 raft_consensus.cc:357] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.348659 1163 raft_consensus.cc:383] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.348745 1163 raft_consensus.cc:738] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.348984 1163 consensus_queue.cc:260] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.349558 1163 ts_tablet_manager.cc:1428] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:15.349869 1163 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.351493 1163 tablet_bootstrap.cc:654] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.353150 1160 tablet_replica.cc:331] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.353319 1160 raft_consensus.cc:2241] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.353487 1160 raft_consensus.cc:2270] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.353957 1160 ts_tablet_manager.cc:1905] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.355762 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f566cc9cdd447798a5cb408ef9cd35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.360900 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 2771159858fc41a1ad930229986edd92 (DEFAULT_TABLE table=default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.363698 850 leader_election.cc:343] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.364539 1160 ts_tablet_manager.cc:1918] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.364346 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2b020ff9f04b4d1796af4ff585cc04aa (table default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]) successfully deleted
W20250625 07:52:15.367599 1268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250625 07:52:15.367883 1018 tablet_service.cc:1468] Processing CreateTablet for tablet be7b755d95b14ecea7a691003455f381 (DEFAULT_TABLE table=default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.368238 1268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:15.368410 1268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 07:52:15.368670 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 154) took 1881 ms. Trace:
I20250625 07:52:15.369442 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.369886 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cbdbd9230210455ba615d51f1a98267b (DEFAULT_TABLE table=default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.372447 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.372815 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.370874 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 541c983d64f348539979f74199f79e59 (DEFAULT_TABLE table=default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.373380 1166 ts_tablet_manager.cc:1918] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.373615 1166 log.cc:1199] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:15.375618 1161 rpcz_store.cc:276] 0625 07:52:13.465897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.465975 (+ 78us) service_pool.cc:225] Handling call
0625 07:52:15.290889 (+1824914us) ts_tablet_manager.cc:1145] Deleting tablet 86ea9df068a94d9187f2f9620e371a39
0625 07:52:15.341857 (+ 50968us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.347151 (+ 5294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809685}
I20250625 07:52:15.369587 1163 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:15.372493 851 leader_election.cc:343] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.380904 1163 ts_tablet_manager.cc:1397] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20250625 07:52:15.380951 851 leader_election.cc:304] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.381784 1167 raft_consensus.cc:2747] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:15.382332 851 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.382404 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec29e4294864fbabdff5a04dcf5b163. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.382261 1163 raft_consensus.cc:357] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.382557 1163 raft_consensus.cc:383] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:15.382613 1268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250625 07:52:15.382645 1163 raft_consensus.cc:738] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:15.382766 1268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250625 07:52:15.382767 1166 ts_tablet_manager.cc:1939] T 239e99c6b91d41d68a213a4c8d79470f P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:15.382930 1268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20250625 07:52:15.382875 1163 consensus_queue.cc:260] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:15.383036 1268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:15.383119 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 55d190e9d11c4589a82fcc7307e9eccb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.383426 1163 ts_tablet_manager.cc:1428] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:15.383718 1163 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.385515 1163 tablet_bootstrap.cc:654] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.385699 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.386116 1020 tablet_service.cc:1468] Processing CreateTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 (DEFAULT_TABLE table=default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.386391 1160 log.cc:1199] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:15.386828 1160 ts_tablet_manager.cc:1939] T 6fc7c0feb0e04ef28c323323e3007efc P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.387715 1160 tablet_replica.cc:331] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.387965 1160 raft_consensus.cc:2241] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.388144 1160 raft_consensus.cc:2270] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.388461 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 239e99c6b91d41d68a213a4c8d79470f (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:15.388770 1160 ts_tablet_manager.cc:1905] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.391182 1167 raft_consensus.cc:491] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.391366 1167 raft_consensus.cc:513] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.391991 1167 leader_election.cc:290] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:15.392679 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 156) took 1890 ms. Trace:
I20250625 07:52:15.394078 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0735d576f947d395bee8d1b865f2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.397509 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1eab049ad7fc4b5581e5c31a14491306" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:15.398823 851 leader_election.cc:343] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.399685 1268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=39255
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
W20250625 07:52:15.399719 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 366d3e6fcc9d49be8fbd2cd070632861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 366d3e6fcc9d49be8fbd2cd070632861 already in progress: creating tablet
I20250625 07:52:15.400794 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.402063 1160 ts_tablet_manager.cc:1918] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.402416 1268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:15.402715 1160 log.cc:1199] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d51a898202154881ab4ad41da1d519a3
I20250625 07:52:15.403259 1160 ts_tablet_manager.cc:1939] T d51a898202154881ab4ad41da1d519a3 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.404579 1160 tablet_replica.cc:331] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.404654 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2771159858fc41a1ad930229986edd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.404999 1268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 07:52:15.405879 1160 raft_consensus.cc:2241] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.406869 1160 raft_consensus.cc:2270] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.409201 1158 tablet_bootstrap.cc:654] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:15.412680 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f9a94dc1432d4cbdbf3eecacb1429c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9a94dc1432d4cbdbf3eecacb1429c23 already in progress: creating tablet
W20250625 07:52:15.413362 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a0518fbc7f52454296ae79a3c92ef800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0518fbc7f52454296ae79a3c92ef800 already in progress: creating tablet
I20250625 07:52:15.414294 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743123ffcf8240dc82bea7178559b7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.413026 1161 rpcz_store.cc:276] 0625 07:52:13.494135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.495250 (+ 1115us) service_pool.cc:225] Handling call
0625 07:52:15.375986 (+1880736us) ts_tablet_manager.cc:1145] Deleting tablet de9ccb0a2d3846228acbc2769786a059
0625 07:52:15.380713 (+ 4727us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.384635 (+ 3922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880524}
I20250625 07:52:15.417245 1160 ts_tablet_manager.cc:1905] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.421087 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5be7c764e2274992a4c2809f83a3dcea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.421352 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2e804b4479be4226bc14ca030cdb1144" } modify_external_catalogs: true
I20250625 07:52:15.421588 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2e804b4479be4226bc14ca030cdb1144" } modify_external_catalogs: true
W20250625 07:52:15.421733 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5be7c764e2274992a4c2809f83a3dcea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.423693 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_38b762c38ec54ae09b10d5524e53a950" } modify_external_catalogs: true
I20250625 07:52:15.423904 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_38b762c38ec54ae09b10d5524e53a950" } modify_external_catalogs: true
I20250625 07:52:15.424221 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8b69c49b5fd54a4dacdae2d2792e230d" } new_table_name: "default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc" modify_external_catalogs: true
I20250625 07:52:15.424612 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cb899cd4017844fa834aafebadd0af7c" } new_table_name: "default.Table_a8adac4095f54fafa0b25109890670ec" modify_external_catalogs: true
I20250625 07:52:15.425778 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dffc5f414b814cf9b044d071025a86cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.425920 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0d054b4221a5407c868cac2f0d7cc131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.426294 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dffc5f414b814cf9b044d071025a86cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:15.426399 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d054b4221a5407c868cac2f0d7cc131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.431985 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541c983d64f348539979f74199f79e59. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.436151 1326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:15.439644 1327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:15.440044 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 155) took 1926 ms. Trace:
I20250625 07:52:15.440716 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (table default.Table_3c1cbfecab414abaa74c66f1773f817e [id=5174ca174ab24d7695eb1fddc9414c74]). Expected version 3 got 0
I20250625 07:52:15.440933 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 365ef91e7bac43e283602d3d036853ff (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]). Expected version 2 got 0
I20250625 07:52:15.441172 589 catalog_manager.cc:5582] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } }
I20250625 07:52:15.443517 1161 rpcz_store.cc:276] 0625 07:52:13.493982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.495494 (+ 1512us) service_pool.cc:225] Handling call
0625 07:52:15.415357 (+1919863us) ts_tablet_manager.cc:1145] Deleting tablet 1e6c2988baa14cbc8c878f79e942880d
0625 07:52:15.415372 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.420202 (+ 4830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919692}
I20250625 07:52:15.444048 1161 tablet_replica.cc:331] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:15.444314 1161 raft_consensus.cc:2241] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.444514 1161 raft_consensus.cc:2270] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.445106 1161 ts_tablet_manager.cc:1905] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.446135 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1ddc3593237d4cfea5aada771b9228d5" } new_table_name: "default.Table_3c133d08fab2423c90088640b5732cab" modify_external_catalogs: true
I20250625 07:52:15.449354 1050 raft_consensus.cc:2466] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:15.450171 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7b755d95b14ecea7a691003455f381. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.452066 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9f4359fd842a4760bae75169e8914f1c (DEFAULT_TABLE table=default.Table_5117224af8bf44ee866ec4497721cf63 [id=914805a4f6684943a2e76c948377f326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.453531 1160 ts_tablet_manager.cc:1918] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:15.454746 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d8d3300befc4f419070ae84725ee96a consensus state has no leader
I20250625 07:52:15.455267 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2131dbf2d90d4c45807fef063380c5ee" } new_table_name: "default.Table_ce2222183ca64261b57847f5be67d81a" modify_external_catalogs: true
I20250625 07:52:15.457410 754 tablet_service.cc:1468] Processing CreateTablet for tablet 1afe90251c964e369d05e82c7db19326 (DEFAULT_TABLE table=default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.458055 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbdbd9230210455ba615d51f1a98267b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.458629 758 tablet_service.cc:1468] Processing CreateTablet for tablet 1898ff5b7626478bb997f7a433477498 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.458825 1160 log.cc:1199] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3338d1c708b4421c8501740da89afec5
I20250625 07:52:15.459239 1160 ts_tablet_manager.cc:1939] T 3338d1c708b4421c8501740da89afec5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.459951 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4359fd842a4760bae75169e8914f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.462457 1160 tablet_replica.cc:331] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:15.463058 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f191436f3b9244eaa12d211b6f125175 consensus state has no leader
I20250625 07:52:15.463424 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_58cafcacc8aa4f77a618e6e3f111d687" } new_table_name: "default.Table_69159065e6254990b2bebfab94d2fd0b" modify_external_catalogs: true
I20250625 07:52:15.467065 1160 raft_consensus.cc:2241] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.467434 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6c3be49be14ecbb3c26ef107ede015. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.467643 1160 raft_consensus.cc:2270] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.468218 1160 ts_tablet_manager.cc:1905] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.468767 749 tablet_service.cc:1468] Processing CreateTablet for tablet c5fe3ea5745743019d23d49ff808070a (DEFAULT_TABLE table=default.Table_9097898be312417e8b4a456060da3a49 [id=c48e5060ff8144d085c60a7790e55fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.469365 746 tablet_service.cc:1468] Processing CreateTablet for tablet cd728ffc2a274deea3fad533d1eaa6da (DEFAULT_TABLE table=default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.470053 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (DEFAULT_TABLE table=default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.470654 756 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3c54204d5446c688cb123edbd430c1 (DEFAULT_TABLE table=default.Table_e498d11f2eb04d23b580c78f5e2f3fcf [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.471717 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 (DEFAULT_TABLE table=default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.472088 750 tablet_service.cc:1468] Processing CreateTablet for tablet f191436f3b9244eaa12d211b6f125175 (DEFAULT_TABLE table=default.Table_cb899cd4017844fa834aafebadd0af7c [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.472455 1280 raft_consensus.cc:491] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.472671 1280 raft_consensus.cc:513] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:15.475525 986 leader_election.cc:343] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:15.478235 747 tablet_service.cc:1468] Processing CreateTablet for tablet 3b9491eb269948a29888a7301ea75d42 (DEFAULT_TABLE table=default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.478892 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 450727d8f33e4eb489969b7974cdeeb1 (table default.Table_5c23014bc5574cc9a64e940e87188050 [id=01cb78211cdc4a9a9b493e31f8c44a99], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 450727d8f33e4eb489969b7974cdeeb1 consensus state has no leader
I20250625 07:52:15.479527 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afe90251c964e369d05e82c7db19326. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.480938 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd728ffc2a274deea3fad533d1eaa6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.484874 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe3ea5745743019d23d49ff808070a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.484885 1160 ts_tablet_manager.cc:1918] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.485030 1160 log.cc:1199] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a558e0a5403d4489892dfce9587e090c
I20250625 07:52:15.485404 1160 ts_tablet_manager.cc:1939] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.486210 1160 tablet_replica.cc:331] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.486392 1160 raft_consensus.cc:2241] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.486565 1160 raft_consensus.cc:2270] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.487104 1160 ts_tablet_manager.cc:1905] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:15.488123 985 leader_election.cc:343] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:15.489185 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab38e0cdcd14f209e6877f2e4a46e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.493361 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3c54204d5446c688cb123edbd430c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.496964 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e60d4f9aed44082b9905fbb44ffb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.501691 1280 leader_election.cc:290] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:15.503800 1315 raft_consensus.cc:2747] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.504272 1160 ts_tablet_manager.cc:1918] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.508282 1160 log.cc:1199] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/09b3830bd506443ab6f06ddd3c6b9974
I20250625 07:52:15.508865 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1898ff5b7626478bb997f7a433477498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.509070 1160 ts_tablet_manager.cc:1939] T 09b3830bd506443ab6f06ddd3c6b9974 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.510308 1160 tablet_replica.cc:331] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.510538 1160 raft_consensus.cc:2241] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.510718 1160 raft_consensus.cc:2270] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.511320 1160 ts_tablet_manager.cc:1905] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.522794 985 leader_election.cc:304] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:15.523296 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b9491eb269948a29888a7301ea75d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.526268 1160 ts_tablet_manager.cc:1918] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.528010 1160 log.cc:1199] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:15.528635 1160 ts_tablet_manager.cc:1939] T 1222f2c1027b42c6ad9627359a93d678 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.530099 1160 tablet_replica.cc:331] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.531397 1160 raft_consensus.cc:2241] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.531898 1160 raft_consensus.cc:2270] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.532775 1160 ts_tablet_manager.cc:1905] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.533300 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 (DEFAULT_TABLE table=default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.533820 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7480c07af7ce4a3889bd8d42d7419d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.539525 1315 raft_consensus.cc:2747] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.539880 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f191436f3b9244eaa12d211b6f125175. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.542126 850 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.542320 850 leader_election.cc:304] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.542754 850 leader_election.cc:304] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7
W20250625 07:52:15.542964 850 raft_consensus.cc:2684] T a558e0a5403d4489892dfce9587e090c P 429026e8d98d46339db143f84897a092: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:15.544925 1315 raft_consensus.cc:491] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.545387 1311 raft_consensus.cc:2747] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.547567 1160 ts_tablet_manager.cc:1918] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.548336 1160 log.cc:1199] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:15.548944 1160 ts_tablet_manager.cc:1939] T b0d31993b4394cfd8832c93ab31b227b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.550390 1160 tablet_replica.cc:331] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.550606 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6fc7c0feb0e04ef28c323323e3007efc (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:15.551600 1160 raft_consensus.cc:2241] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.552088 1160 raft_consensus.cc:2270] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.552901 1160 ts_tablet_manager.cc:1905] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.556735 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d51a898202154881ab4ad41da1d519a3 (table default.Table_9e0159dbcce04493a7cc2849bde83764 [id=21758be8aff9485b9313da6b33405f8e]) successfully deleted
I20250625 07:52:15.557807 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3338d1c708b4421c8501740da89afec5 (table default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]) successfully deleted
I20250625 07:52:15.559110 885 tablet_service.cc:1468] Processing CreateTablet for tablet f191436f3b9244eaa12d211b6f125175 (DEFAULT_TABLE table=default.Table_cb899cd4017844fa834aafebadd0af7c [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.560024 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f191436f3b9244eaa12d211b6f125175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.564841 880 tablet_service.cc:1468] Processing CreateTablet for tablet 1898ff5b7626478bb997f7a433477498 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.564985 1160 ts_tablet_manager.cc:1918] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.565145 1160 log.cc:1199] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/86ea9df068a94d9187f2f9620e371a39
I20250625 07:52:15.565318 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1898ff5b7626478bb997f7a433477498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.565555 1160 ts_tablet_manager.cc:1939] T 86ea9df068a94d9187f2f9620e371a39 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.566553 1162 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:15.566560 1160 tablet_replica.cc:331] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.566695 1162 ts_tablet_manager.cc:1397] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.526s user 0.009s sys 0.001s
I20250625 07:52:15.566771 1160 raft_consensus.cc:2241] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.566970 1160 raft_consensus.cc:2270] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.567452 1160 ts_tablet_manager.cc:1905] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.567520 1162 raft_consensus.cc:357] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.567687 1162 raft_consensus.cc:383] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.567776 1162 raft_consensus.cc:738] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.568029 1162 consensus_queue.cc:260] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.570943 1311 raft_consensus.cc:491] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.571180 1311 raft_consensus.cc:513] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.571556 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a558e0a5403d4489892dfce9587e090c (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
W20250625 07:52:15.571848 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c44b0ef2ac60457bb56abead72f0182a because the tablet was not found. No further retry: Not found: Tablet not found: c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:15.572074 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 09b3830bd506443ab6f06ddd3c6b9974 (table default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]) successfully deleted
W20250625 07:52:15.572276 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7db110bb6cc146a393d62ddabbe1955a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db110bb6cc146a393d62ddabbe1955a already in progress: creating tablet
I20250625 07:52:15.572499 1311 leader_election.cc:290] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:15.572680 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2db9e8d0fefa47bfaeee462dba096b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db9e8d0fefa47bfaeee462dba096b49 already in progress: creating tablet
I20250625 07:52:15.572928 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1222f2c1027b42c6ad9627359a93d678 (table default.Table_3c1cbfecab414abaa74c66f1773f817e [id=5174ca174ab24d7695eb1fddc9414c74]) successfully deleted
I20250625 07:52:15.573026 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b71aadf074a4a39943b2ff930909f93" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.573048 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 (DEFAULT_TABLE table=default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.573755 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7480c07af7ce4a3889bd8d42d7419d57. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.573709 851 leader_election.cc:343] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.573838 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b0d31993b4394cfd8832c93ab31b227b (table default.Table_12188272bb8e4c26bb94db8721d37c26 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
W20250625 07:52:15.574059 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 86ea9df068a94d9187f2f9620e371a39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86ea9df068a94d9187f2f9620e371a39 already in progress: creating tablet
W20250625 07:52:15.574694 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet de9ccb0a2d3846228acbc2769786a059 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de9ccb0a2d3846228acbc2769786a059 already in progress: creating tablet
W20250625 07:52:15.575122 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1e6c2988baa14cbc8c878f79e942880d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e6c2988baa14cbc8c878f79e942880d already in progress: creating tablet
I20250625 07:52:15.576610 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 (DEFAULT_TABLE table=default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.577165 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e60d4f9aed44082b9905fbb44ffb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.579854 1315 raft_consensus.cc:513] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.576623 880 tablet_service.cc:1468] Processing CreateTablet for tablet 3b9491eb269948a29888a7301ea75d42 (DEFAULT_TABLE table=default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.581034 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3c54204d5446c688cb123edbd430c1 (DEFAULT_TABLE table=default.Table_e498d11f2eb04d23b580c78f5e2f3fcf [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.581138 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b9491eb269948a29888a7301ea75d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.581496 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3c54204d5446c688cb123edbd430c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.585107 892 tablet_service.cc:1468] Processing CreateTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (DEFAULT_TABLE table=default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.585561 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab38e0cdcd14f209e6877f2e4a46e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.585986 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 (DEFAULT_TABLE table=default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.586874 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 (DEFAULT_TABLE table=default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.587106 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1898ff5b7626478bb997f7a433477498 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.587247 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3b9491eb269948a29888a7301ea75d42 (DEFAULT_TABLE table=default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.582763 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1eab049ad7fc4b5581e5c31a14491306" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.589010 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1afe90251c964e369d05e82c7db19326 (DEFAULT_TABLE table=default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.589375 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afe90251c964e369d05e82c7db19326. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.587357 1022 tablet_service.cc:1468] Processing CreateTablet for tablet f191436f3b9244eaa12d211b6f125175 (DEFAULT_TABLE table=default.Table_cb899cd4017844fa834aafebadd0af7c [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.595199 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9f4359fd842a4760bae75169e8914f1c (DEFAULT_TABLE table=default.Table_5117224af8bf44ee866ec4497721cf63 [id=914805a4f6684943a2e76c948377f326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.596621 1162 ts_tablet_manager.cc:1428] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250625 07:52:15.597843 1162 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:15.599231 1268 server_base.cc:1048] running on GCE node
I20250625 07:52:15.599787 1162 tablet_bootstrap.cc:654] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:15.599623 1330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:15.602506 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4359fd842a4760bae75169e8914f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.602589 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e60d4f9aed44082b9905fbb44ffb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.600921 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1afe90251c964e369d05e82c7db19326 (DEFAULT_TABLE table=default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.603278 1268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:15.605005 895 tablet_service.cc:1468] Processing CreateTablet for tablet cd728ffc2a274deea3fad533d1eaa6da (DEFAULT_TABLE table=default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.605261 850 leader_election.cc:343] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.605406 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd728ffc2a274deea3fad533d1eaa6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.605827 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8b3c54204d5446c688cb123edbd430c1 (DEFAULT_TABLE table=default.Table_e498d11f2eb04d23b580c78f5e2f3fcf [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.606163 850 leader_election.cc:304] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.606550 884 tablet_service.cc:1468] Processing CreateTablet for tablet c5fe3ea5745743019d23d49ff808070a (DEFAULT_TABLE table=default.Table_9097898be312417e8b4a456060da3a49 [id=c48e5060ff8144d085c60a7790e55fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.606660 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ee01c47b2df2467eadab63a9cbfb4b6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.607234 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee01c47b2df2467eadab63a9cbfb4b6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.607225 1311 raft_consensus.cc:2747] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.608953 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4de62938b87249fdb2950c71864eeb4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.609472 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4de62938b87249fdb2950c71864eeb4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.610607 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2771159858fc41a1ad930229986edd92
I20250625 07:52:15.611440 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aec9b6017944d5f8293424556b380d3" } modify_external_catalogs: true
I20250625 07:52:15.611605 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aec9b6017944d5f8293424556b380d3" } modify_external_catalogs: true
I20250625 07:52:15.611862 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe3ea5745743019d23d49ff808070a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.612932 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2a9945f38ef148caac59acaadcb2ef4c
I20250625 07:52:15.613631 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_51623e42db554f96a29174eb61aa970c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.614195 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51623e42db554f96a29174eb61aa970c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.619100 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:15.619870 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2a9945f38ef148caac59acaadcb2ef4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9945f38ef148caac59acaadcb2ef4c already in progress: creating tablet
I20250625 07:52:15.620520 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (DEFAULT_TABLE table=default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.621255 1315 leader_election.cc:290] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:15.623225 1311 raft_consensus.cc:491] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.623390 1311 raft_consensus.cc:513] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.623801 1161 ts_tablet_manager.cc:1918] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.623935 1161 log.cc:1199] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b0d31993b4394cfd8832c93ab31b227b
I20250625 07:52:15.624054 1311 leader_election.cc:290] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:15.624307 1161 ts_tablet_manager.cc:1939] T b0d31993b4394cfd8832c93ab31b227b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:15.625072 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fdb6a689ce6440ab7037484e2b9cdc5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:15.625587 851 leader_election.cc:343] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.624630 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 158) took 2079 ms. Trace:
I20250625 07:52:15.626245 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:15.627993 597 master_service.cc:948] ReplaceTablet: received request to replace tablet a5a3751333bc48fc8a59b0e7b67a9935 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:15.628522 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_098cd87d74e84fe7bd51f4b61ee6ad60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.630698 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_098cd87d74e84fe7bd51f4b61ee6ad60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.630930 1012 tablet_service.cc:1468] Processing CreateTablet for tablet c5fe3ea5745743019d23d49ff808070a (DEFAULT_TABLE table=default.Table_9097898be312417e8b4a456060da3a49 [id=c48e5060ff8144d085c60a7790e55fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.631906 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.632223 1166 tablet_replica.cc:331] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:15.632408 1166 raft_consensus.cc:2241] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.632575 1166 raft_consensus.cc:2270] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.633286 1166 ts_tablet_manager.cc:1905] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.635500 1014 tablet_service.cc:1468] Processing CreateTablet for tablet cd728ffc2a274deea3fad533d1eaa6da (DEFAULT_TABLE table=default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.636098 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 9f4359fd842a4760bae75169e8914f1c (DEFAULT_TABLE table=default.Table_5117224af8bf44ee866ec4497721cf63 [id=914805a4f6684943a2e76c948377f326]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:15.636502 986 leader_election.cc:343] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:15.638964 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b71aadf074a4a39943b2ff930909f93" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:15.639990 850 leader_election.cc:343] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.640163 850 leader_election.cc:304] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:15.640385 589 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 365ef91e7bac43e283602d3d036853ff (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 365ef91e7bac43e283602d3d036853ff consensus state has no leader
I20250625 07:52:15.640599 1311 raft_consensus.cc:2747] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.640960 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:15.641248 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1898ff5b7626478bb997f7a433477498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.641791 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0cfa01917c304be98bc1f8d5fb704b47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.642557 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cfa01917c304be98bc1f8d5fb704b47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.643383 1166 ts_tablet_manager.cc:1918] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.643532 1166 log.cc:1199] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6b3488a9fb19452dac3241170dd2fee2
I20250625 07:52:15.643950 1166 ts_tablet_manager.cc:1939] T 6b3488a9fb19452dac3241170dd2fee2 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:15.644800 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c928e3f5ce01487c976a35b088cbe9ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.645221 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:15.645342 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c928e3f5ce01487c976a35b088cbe9ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.645606 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.646116 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6b3488a9fb19452dac3241170dd2fee2 (table default.Table_e407dc6ea5d247b1a9c26729116da0e2 [id=02ec3b2efaed40e7b6354e1f15d78043]) successfully deleted
I20250625 07:52:15.646126 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fdb6a689ce6440ab7037484e2b9cdc5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:15.646462 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 43648us
W20250625 07:52:15.647256 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 494cacf2dcbb40d3b2c9f05f64809870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 494cacf2dcbb40d3b2c9f05f64809870 already in progress: creating tablet
W20250625 07:52:15.647975 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ab38e0cdcd14f209e6877f2e4a46e98 already in progress: creating tablet
W20250625 07:52:15.648330 850 leader_election.cc:343] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.648509 850 leader_election.cc:304] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.643522 1161 rpcz_store.cc:276] 0625 07:52:13.545599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.546375 (+ 776us) service_pool.cc:225] Handling call
0625 07:52:15.443946 (+1897571us) ts_tablet_manager.cc:1145] Deleting tablet b0d31993b4394cfd8832c93ab31b227b
0625 07:52:15.443973 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.578932 (+134959us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:15.623690 (+ 44758us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:15.624578 (+ 888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1897327}
I20250625 07:52:15.649015 1311 raft_consensus.cc:2747] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:15.649240 985 leader_election.cc:343] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:15.650995 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b9491eb269948a29888a7301ea75d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.655144 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f191436f3b9244eaa12d211b6f125175. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.658706 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcc3afc210a540058294121d2b83c66d consensus state has no leader
I20250625 07:52:15.659147 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_405b7023434c463bbce1552ceb099b75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.659636 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1afe90251c964e369d05e82c7db19326. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.659659 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_405b7023434c463bbce1552ceb099b75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.660497 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b3c54204d5446c688cb123edbd430c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.661648 1163 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.661852 1163 ts_tablet_manager.cc:1397] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.278s user 0.033s sys 0.038s
I20250625 07:52:15.662659 1163 raft_consensus.cc:357] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.662894 1163 raft_consensus.cc:383] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.662971 1163 raft_consensus.cc:738] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.663262 1163 consensus_queue.cc:260] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.663869 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7480c07af7ce4a3889bd8d42d7419d57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.663930 1163 ts_tablet_manager.cc:1428] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:15.664291 1163 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.666175 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:15.668026 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c8c317c10e14332a3b3919808e4a512 consensus state has no leader
W20250625 07:52:15.668676 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02ec8c9e6340497abe802c7345f42921 consensus state has no leader
I20250625 07:52:15.669579 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ab38e0cdcd14f209e6877f2e4a46e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.669976 1163 tablet_bootstrap.cc:654] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:15.671599 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2771159858fc41a1ad930229986edd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2771159858fc41a1ad930229986edd92 already in progress: creating tablet
I20250625 07:52:15.674973 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4d7c95ebabf74ee1a260a5996137c0a0" } new_table_name: "default.Table_a881661cd92a4541a73b939c4532acc7" modify_external_catalogs: true
I20250625 07:52:15.675140 1160 ts_tablet_manager.cc:1918] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.675259 1160 log.cc:1199] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6b3488a9fb19452dac3241170dd2fee2
I20250625 07:52:15.675613 1160 ts_tablet_manager.cc:1939] T 6b3488a9fb19452dac3241170dd2fee2 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.676553 1160 tablet_replica.cc:331] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.676568 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a3226ecfd38140ef85bca57822e323e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.677222 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3226ecfd38140ef85bca57822e323e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.676925 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.678372 1160 raft_consensus.cc:2241] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.678618 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b14bab363cb9435f86290aa16ed47a6c" } new_table_name: "default.Table_4e9b498709a94e19a3f242fad2d3abb2" modify_external_catalogs: true
I20250625 07:52:15.678982 1160 raft_consensus.cc:2270] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.679764 1160 ts_tablet_manager.cc:1905] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.679919 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4359fd842a4760bae75169e8914f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.679769 985 leader_election.cc:304] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:15.684340 1315 raft_consensus.cc:2747] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.689635 1161 tablet_replica.cc:331] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:15.690572 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c81396ee6ed7400baa30dcc5d71ba45d
I20250625 07:52:15.691371 1311 consensus_queue.cc:1035] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:15.692992 1160 ts_tablet_manager.cc:1918] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.693127 1160 log.cc:1199] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:15.693531 1160 ts_tablet_manager.cc:1939] T 239e99c6b91d41d68a213a4c8d79470f P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:15.696943 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 82820.7us
I20250625 07:52:15.698285 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.699110 1161 raft_consensus.cc:2241] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:15.699524 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c81396ee6ed7400baa30dcc5d71ba45d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c81396ee6ed7400baa30dcc5d71ba45d already in progress: creating tablet
I20250625 07:52:15.699965 1160 tablet_replica.cc:331] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.700444 1160 raft_consensus.cc:2241] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.700743 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd728ffc2a274deea3fad533d1eaa6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.700901 1160 raft_consensus.cc:2270] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.701970 1160 ts_tablet_manager.cc:1905] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.712924 1161 raft_consensus.cc:2270] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.713516 1161 ts_tablet_manager.cc:1905] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.714020 1163 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.714185 1163 ts_tablet_manager.cc:1397] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.050s user 0.018s sys 0.000s
I20250625 07:52:15.714782 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe3ea5745743019d23d49ff808070a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.715107 1163 raft_consensus.cc:357] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.715291 1163 raft_consensus.cc:383] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.715382 1163 raft_consensus.cc:738] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.715622 1163 consensus_queue.cc:260] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.716984 1163 ts_tablet_manager.cc:1428] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:15.717841 1163 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.719723 1163 tablet_bootstrap.cc:654] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.721148 1160 ts_tablet_manager.cc:1918] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.721372 1160 log.cc:1199] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/23755f70822740e08ca6a5e5b5571458
I20250625 07:52:15.721861 1160 ts_tablet_manager.cc:1939] T 23755f70822740e08ca6a5e5b5571458 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.724355 1160 tablet_replica.cc:331] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.724887 1160 raft_consensus.cc:2241] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.725443 1160 raft_consensus.cc:2270] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.726358 1160 ts_tablet_manager.cc:1905] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.727525 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.728135 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.728435 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.729032 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.730798 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 86ea9df068a94d9187f2f9620e371a39 (table default.Table_17d5751097fa40feaf90d0fb3c97e1b5 [id=5281aca609cf421990f96de3424b4442]) successfully deleted
I20250625 07:52:15.732074 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6b3488a9fb19452dac3241170dd2fee2 (table default.Table_e407dc6ea5d247b1a9c26729116da0e2 [id=02ec3b2efaed40e7b6354e1f15d78043]) successfully deleted
I20250625 07:52:15.732370 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 239e99c6b91d41d68a213a4c8d79470f (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:15.732735 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 23755f70822740e08ca6a5e5b5571458 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
W20250625 07:52:15.733042 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f9a94dc1432d4cbdbf3eecacb1429c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9a94dc1432d4cbdbf3eecacb1429c23 already in progress: creating tablet
W20250625 07:52:15.733461 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 366d3e6fcc9d49be8fbd2cd070632861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 366d3e6fcc9d49be8fbd2cd070632861 already in progress: creating tablet
I20250625 07:52:15.736502 1160 ts_tablet_manager.cc:1918] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.740193 1163 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.740408 1163 ts_tablet_manager.cc:1397] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.002s
I20250625 07:52:15.741331 1163 raft_consensus.cc:357] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.741577 1163 raft_consensus.cc:383] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.741662 1163 raft_consensus.cc:738] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.741961 1163 consensus_queue.cc:260] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.742581 1163 ts_tablet_manager.cc:1428] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:15.743145 1163 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.745327 1163 tablet_bootstrap.cc:654] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.746477 1160 log.cc:1199] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b7139335c7124ab79bc2d0a251984199
I20250625 07:52:15.746953 1160 ts_tablet_manager.cc:1939] T b7139335c7124ab79bc2d0a251984199 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.750301 1160 tablet_replica.cc:331] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.750735 1160 raft_consensus.cc:2241] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.751196 1160 raft_consensus.cc:2270] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.752877 1160 ts_tablet_manager.cc:1905] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.765192 744 tablet_service.cc:1468] Processing CreateTablet for tablet c934a46a8f6c407caa7a84d6415d6901 (DEFAULT_TABLE table=default.Table_5be7c764e2274992a4c2809f83a3dcea [id=6889b79d2a004cf894a70fbe2402f7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.765738 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934a46a8f6c407caa7a84d6415d6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.769928 750 tablet_service.cc:1468] Processing CreateTablet for tablet 0929ed76622248d1a41f08f4258108cd (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.770378 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0929ed76622248d1a41f08f4258108cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.776263 747 tablet_service.cc:1468] Processing CreateTablet for tablet f7f82e83ed924d2bbedd33d96642a10a (DEFAULT_TABLE table=default.Table_2a4d9d86ac944371a48bcdd835f6c025 [id=26371e29bd6e46ef86cd7db4c00bde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.776679 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f82e83ed924d2bbedd33d96642a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.782373 1163 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.782622 1163 ts_tablet_manager.cc:1397] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20250625 07:52:15.783500 1163 raft_consensus.cc:357] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.783766 1163 raft_consensus.cc:383] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.783931 1163 raft_consensus.cc:738] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.784245 1163 consensus_queue.cc:260] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.785518 1163 ts_tablet_manager.cc:1428] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:15.785925 1163 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.788062 758 tablet_service.cc:1468] Processing CreateTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 (DEFAULT_TABLE table=default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.788404 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.788487 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289ea5553ac4ff983871ece4fc72ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.795468 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8cadd41655404316b10939e46d370c76 (DEFAULT_TABLE table=default.Table_210190c86c3c4587986ad321a3d67e0b [id=03e6478224ea4f819a9533dd713b3bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.795732 1308 consensus_queue.cc:1035] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:15.795877 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cadd41655404316b10939e46d370c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.798874 1163 tablet_bootstrap.cc:654] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.799434 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.799866 1160 ts_tablet_manager.cc:1918] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.800055 1160 log.cc:1199] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/012df7beedb1446ca5e533d6624afc10
I20250625 07:52:15.800473 1160 ts_tablet_manager.cc:1939] T 012df7beedb1446ca5e533d6624afc10 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.804651 747 tablet_service.cc:1468] Processing CreateTablet for tablet a1baf94a051e485eaf4c0e074e147252 (DEFAULT_TABLE table=default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.805632 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1baf94a051e485eaf4c0e074e147252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.806600 1163 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.806833 1163 ts_tablet_manager.cc:1397] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250625 07:52:15.808076 1163 raft_consensus.cc:357] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.808760 1163 raft_consensus.cc:383] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.809028 1163 raft_consensus.cc:738] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.809350 1163 consensus_queue.cc:260] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.809953 1163 ts_tablet_manager.cc:1428] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:15.810616 1163 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.812529 1163 tablet_bootstrap.cc:654] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.824558 1163 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.824842 1163 ts_tablet_manager.cc:1397] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20250625 07:52:15.825754 1163 raft_consensus.cc:357] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.826059 1163 raft_consensus.cc:383] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.826210 1163 raft_consensus.cc:738] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.826568 1163 consensus_queue.cc:260] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.827669 1163 ts_tablet_manager.cc:1428] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:15.828447 1163 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.829466 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.831313 1163 tablet_bootstrap.cc:654] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.839486 1163 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.839694 1163 ts_tablet_manager.cc:1397] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250625 07:52:15.840435 1163 raft_consensus.cc:357] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.840727 1163 raft_consensus.cc:383] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.840849 1163 raft_consensus.cc:738] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.841040 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.841130 1163 consensus_queue.cc:260] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.842308 1163 ts_tablet_manager.cc:1428] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:15.843076 1163 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.845371 1163 tablet_bootstrap.cc:654] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.847805 1158 tablet_bootstrap.cc:492] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:15.847988 1158 ts_tablet_manager.cc:1397] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.564s user 0.027s sys 0.030s
I20250625 07:52:15.849699 1158 raft_consensus.cc:357] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.849885 1158 raft_consensus.cc:383] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.849965 1158 raft_consensus.cc:738] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.738095 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 250175f54ba94a9fbc2455d9cf27fb2e from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:15.850250 1158 consensus_queue.cc:260] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:15.750427 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3670cadfa2841949705c65c9b6c0927 consensus state has no leader
W20250625 07:52:15.751093 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4c744fef0e24cb39c4bb0787adb1cbc consensus state has no leader
I20250625 07:52:15.817209 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ddae42a67be34593a367bf6f3d9e8cde" } modify_external_catalogs: true
I20250625 07:52:15.850965 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ddae42a67be34593a367bf6f3d9e8cde" } modify_external_catalogs: true
I20250625 07:52:15.817744 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e75429a686be4882a9676251755c548b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.852155 1158 ts_tablet_manager.cc:1428] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250625 07:52:15.852211 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e75429a686be4882a9676251755c548b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.852514 1158 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:15.818110 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_24edc711d8fd46d7b8d3e7d20be99016"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.853389 1311 raft_consensus.cc:491] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.853551 1311 raft_consensus.cc:513] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:15.853786 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24edc711d8fd46d7b8d3e7d20be99016 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.854192 1311 leader_election.cc:290] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:15.818421 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_27c3e9984caf46e0ac463d7dc5b7edc4" } modify_external_catalogs: true
I20250625 07:52:15.831413 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_339dbe043bad47148d466ca915d84d33" } modify_external_catalogs: true
I20250625 07:52:15.854938 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_27c3e9984caf46e0ac463d7dc5b7edc4" } modify_external_catalogs: true
I20250625 07:52:15.854972 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_339dbe043bad47148d466ca915d84d33" } modify_external_catalogs: true
I20250625 07:52:15.855082 1163 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.855049 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "365ef91e7bac43e283602d3d036853ff" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.855239 1163 ts_tablet_manager.cc:1397] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250625 07:52:15.844126 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce2222183ca64261b57847f5be67d81a" } modify_external_catalogs: true
I20250625 07:52:15.855490 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce2222183ca64261b57847f5be67d81a" } modify_external_catalogs: true
W20250625 07:52:15.855665 850 leader_election.cc:343] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.855971 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "365ef91e7bac43e283602d3d036853ff" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:15.856051 1163 raft_consensus.cc:357] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.856230 1163 raft_consensus.cc:383] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.856323 1163 raft_consensus.cc:738] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:15.856554 851 leader_election.cc:343] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:15.856680 851 leader_election.cc:304] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:15.856962 1311 raft_consensus.cc:2747] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:15.856572 1163 consensus_queue.cc:260] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:15.857982 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 124507us
I20250625 07:52:15.860018 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.860747 884 tablet_service.cc:1468] Processing CreateTablet for tablet c934a46a8f6c407caa7a84d6415d6901 (DEFAULT_TABLE table=default.Table_5be7c764e2274992a4c2809f83a3dcea [id=6889b79d2a004cf894a70fbe2402f7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.861194 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934a46a8f6c407caa7a84d6415d6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.861400 1163 ts_tablet_manager.cc:1428] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:15.861726 1163 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.862367 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0929ed76622248d1a41f08f4258108cd (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.862749 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0929ed76622248d1a41f08f4258108cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.864444 1163 tablet_bootstrap.cc:654] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.865010 889 tablet_service.cc:1468] Processing CreateTablet for tablet f7f82e83ed924d2bbedd33d96642a10a (DEFAULT_TABLE table=default.Table_2a4d9d86ac944371a48bcdd835f6c025 [id=26371e29bd6e46ef86cd7db4c00bde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.865242 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9097898be312417e8b4a456060da3a49" } new_table_name: "default.Table_f9a2ecb876ab4b089993854b257c3450" modify_external_catalogs: true
I20250625 07:52:15.865378 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f82e83ed924d2bbedd33d96642a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.869025 890 tablet_service.cc:1468] Processing CreateTablet for tablet a1baf94a051e485eaf4c0e074e147252 (DEFAULT_TABLE table=default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.869406 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 239e99c6b91d41d68a213a4c8d79470f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:15.869462 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1baf94a051e485eaf4c0e074e147252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.871773 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dffc5f414b814cf9b044d071025a86cd" } new_table_name: "default.Table_7bdec7af96c94b47a5b5c890625f02cc" modify_external_catalogs: true
I20250625 07:52:15.872828 1160 tablet_replica.cc:331] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.873044 1160 raft_consensus.cc:2241] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.872995 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0929ed76622248d1a41f08f4258108cd (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.873225 1160 raft_consensus.cc:2270] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.873172 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c133d08fab2423c90088640b5732cab" } new_table_name: "default.Table_b167db2ea8944d13b34adbb44e815b62" modify_external_catalogs: true
I20250625 07:52:15.873385 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0929ed76622248d1a41f08f4258108cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.873629 887 tablet_service.cc:1468] Processing CreateTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 (DEFAULT_TABLE table=default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.873713 1158 tablet_bootstrap.cc:654] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.873991 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289ea5553ac4ff983871ece4fc72ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.874729 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1a536939b2ae4f5b9dc8b8bc27da686f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.875272 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a536939b2ae4f5b9dc8b8bc27da686f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.876511 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:15.877302 895 tablet_service.cc:1468] Processing CreateTablet for tablet 8cadd41655404316b10939e46d370c76 (DEFAULT_TABLE table=default.Table_210190c86c3c4587986ad321a3d67e0b [id=03e6478224ea4f819a9533dd713b3bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.877635 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cadd41655404316b10939e46d370c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.879887 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b7139335c7124ab79bc2d0a251984199 (table default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0 [id=84244d54b35c47829f8f8a92c3d47ee3]) successfully deleted
W20250625 07:52:15.880100 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a0518fbc7f52454296ae79a3c92ef800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0518fbc7f52454296ae79a3c92ef800 already in progress: creating tablet
W20250625 07:52:15.880277 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2771159858fc41a1ad930229986edd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2771159858fc41a1ad930229986edd92 already in progress: creating tablet
W20250625 07:52:15.880434 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2a9945f38ef148caac59acaadcb2ef4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9945f38ef148caac59acaadcb2ef4c already in progress: creating tablet
W20250625 07:52:15.880600 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ab38e0cdcd14f209e6877f2e4a46e98 already in progress: creating tablet
I20250625 07:52:15.880817 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 012df7beedb1446ca5e533d6624afc10 (table default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]) successfully deleted
W20250625 07:52:15.881038 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 494cacf2dcbb40d3b2c9f05f64809870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 494cacf2dcbb40d3b2c9f05f64809870 already in progress: creating tablet
I20250625 07:52:15.881567 1014 tablet_service.cc:1468] Processing CreateTablet for tablet c934a46a8f6c407caa7a84d6415d6901 (DEFAULT_TABLE table=default.Table_5be7c764e2274992a4c2809f83a3dcea [id=6889b79d2a004cf894a70fbe2402f7b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.881811 1160 ts_tablet_manager.cc:1905] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.882050 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c934a46a8f6c407caa7a84d6415d6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.883399 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_817356d9efcf4f889897f9e942387859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.883976 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_817356d9efcf4f889897f9e942387859 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.888249 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0f566cc9cdd447798a5cb408ef9cd35a
I20250625 07:52:15.889120 1161 ts_tablet_manager.cc:1918] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.889349 1161 log.cc:1199] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3e14b60ee7a843e5bb878148d8fd2e53
I20250625 07:52:15.889794 1161 ts_tablet_manager.cc:1939] T 3e14b60ee7a843e5bb878148d8fd2e53 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:15.890720 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 170) took 2233 ms. Trace:
I20250625 07:52:15.890861 1161 rpcz_store.cc:276] 0625 07:52:13.657361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.657433 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:15.653577 (+1996144us) ts_tablet_manager.cc:1145] Deleting tablet 3e14b60ee7a843e5bb878148d8fd2e53
0625 07:52:15.653589 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.817348 (+163759us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:15.888970 (+ 71622us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:15.890671 (+ 1701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969808}
W20250625 07:52:15.898543 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5fe3ea5745743019d23d49ff808070a consensus state has no leader
I20250625 07:52:15.901594 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5c23014bc5574cc9a64e940e87188050" } modify_external_catalogs: true
I20250625 07:52:15.901800 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5c23014bc5574cc9a64e940e87188050" } modify_external_catalogs: true
W20250625 07:52:15.906733 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4627f4da6b4542aff1e44157087b14 has no consensus state
I20250625 07:52:15.911939 595 master_service.cc:948] ReplaceTablet: received request to replace tablet d51a898202154881ab4ad41da1d519a3 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:15.914302 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_15745c1b68f54870b34b4ced0001369f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.914912 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15745c1b68f54870b34b4ced0001369f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.916358 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3200657505f94bf7a15245040bb3b059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.916939 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3200657505f94bf7a15245040bb3b059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.918781 1160 ts_tablet_manager.cc:1918] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:15.918992 1160 log.cc:1199] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c81396ee6ed7400baa30dcc5d71ba45d
I20250625 07:52:15.919477 1160 ts_tablet_manager.cc:1939] T c81396ee6ed7400baa30dcc5d71ba45d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:15.921914 1162 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:15.922324 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcc3afc210a540058294121d2b83c66d consensus state has no leader
I20250625 07:52:15.922286 1162 ts_tablet_manager.cc:1397] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.325s user 0.004s sys 0.018s
I20250625 07:52:15.922605 594 master_service.cc:948] ReplaceTablet: received request to replace tablet a777f64425384be5bccc86b197626618 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:15.924321 1162 raft_consensus.cc:357] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.924703 1162 raft_consensus.cc:383] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.924858 1162 raft_consensus.cc:738] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.925218 1162 consensus_queue.cc:260] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:15.926097 1162 ts_tablet_manager.cc:1428] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:15.930557 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 6b71aadf074a4a39943b2ff930909f93 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]). Expected version 1 got 0
I20250625 07:52:15.930770 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 32bd7cdef96946e88683e457c2f26c9b (table default.Table_aac2883a6c934d2aa9de60cf6f345fad [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 1 got 0
I20250625 07:52:15.933022 1162 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:15.940943 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a48b063f27e347c2974134be14e26739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:15.941517 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a48b063f27e347c2974134be14e26739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.937206 750 tablet_service.cc:1468] Processing CreateTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 (DEFAULT_TABLE table=default.Table_ee01c47b2df2467eadab63a9cbfb4b6a [id=860ff357dc3a4a2dad9d850c113cf464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.943377 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8e8ec360a4e6476db25022ee41cf9eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:15.943892 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50c35cac8e94e088cdacb664b8adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:15.943938 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e8ec360a4e6476db25022ee41cf9eba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:15.938058 758 tablet_service.cc:1468] Processing CreateTablet for tablet 871b603cb92249c6b0f16c825bf46232 (DEFAULT_TABLE table=default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.945019 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871b603cb92249c6b0f16c825bf46232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.938666 756 tablet_service.cc:1468] Processing CreateTablet for tablet 019959e9abed46fbbbb4755cb01cef44 (DEFAULT_TABLE table=default.Table_c928e3f5ce01487c976a35b088cbe9ad [id=ea62e41d8d904cf5ac63b9d0a62cde09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.949051 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019959e9abed46fbbbb4755cb01cef44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.894372 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f7f82e83ed924d2bbedd33d96642a10a (DEFAULT_TABLE table=default.Table_2a4d9d86ac944371a48bcdd835f6c025 [id=26371e29bd6e46ef86cd7db4c00bde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.953321 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7f82e83ed924d2bbedd33d96642a10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.953378 1162 tablet_bootstrap.cc:654] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:15.953943 1268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:15.897172 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a1baf94a051e485eaf4c0e074e147252 (DEFAULT_TABLE table=default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.957367 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1baf94a051e485eaf4c0e074e147252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.939417 744 tablet_service.cc:1468] Processing CreateTablet for tablet b2bb060cdd9a48688967989d7491cfc1 (DEFAULT_TABLE table=default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.898859 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8cadd41655404316b10939e46d370c76 (DEFAULT_TABLE table=default.Table_210190c86c3c4587986ad321a3d67e0b [id=03e6478224ea4f819a9533dd713b3bc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.902690 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 (DEFAULT_TABLE table=default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.959228 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cadd41655404316b10939e46d370c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.960321 746 tablet_service.cc:1468] Processing CreateTablet for tablet ccddd630a82341cba7a857e67a349fe0 (DEFAULT_TABLE table=default.Table_4de62938b87249fdb2950c71864eeb4c [id=3a15e8259a5f4b1283f0c106d55baff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.960743 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccddd630a82341cba7a857e67a349fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.964841 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e289ea5553ac4ff983871ece4fc72ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.969960 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bb060cdd9a48688967989d7491cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.970729 745 tablet_service.cc:1468] Processing CreateTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (DEFAULT_TABLE table=default.Table_098cd87d74e84fe7bd51f4b61ee6ad60 [id=3dcf0706033c48ceaa1fdd5763284bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.971043 749 tablet_service.cc:1468] Processing CreateTablet for tablet 4b4627f4da6b4542aff1e44157087b14 (DEFAULT_TABLE table=default.Table_7bdec7af96c94b47a5b5c890625f02cc [id=618d684f882f425dab0d9a7395deb004]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.971257 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2f72b227e14e46983c906520c29fa3 (DEFAULT_TABLE table=default.Table_51623e42db554f96a29174eb61aa970c [id=2bdf3240cc0946c5a47a7232d5a200db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.971873 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.975064 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4627f4da6b4542aff1e44157087b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.929260 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.978962 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.978926 892 tablet_service.cc:1468] Processing CreateTablet for tablet 4167282133f94bfb9870bd08fc6594b4 (DEFAULT_TABLE table=default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.979115 882 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2f72b227e14e46983c906520c29fa3 (DEFAULT_TABLE table=default.Table_51623e42db554f96a29174eb61aa970c [id=2bdf3240cc0946c5a47a7232d5a200db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.979561 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4167282133f94bfb9870bd08fc6594b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.980800 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:15.981032 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2f72b227e14e46983c906520c29fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.981211 1160 tablet_replica.cc:331] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:15.981452 1160 raft_consensus.cc:2241] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:15.981484 890 tablet_service.cc:1468] Processing CreateTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 (DEFAULT_TABLE table=default.Table_ee01c47b2df2467eadab63a9cbfb4b6a [id=860ff357dc3a4a2dad9d850c113cf464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.981649 1160 raft_consensus.cc:2270] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:15.981871 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50c35cac8e94e088cdacb664b8adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.982241 1160 ts_tablet_manager.cc:1905] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:15.981251 884 tablet_service.cc:1468] Processing CreateTablet for tablet b2bb060cdd9a48688967989d7491cfc1 (DEFAULT_TABLE table=default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.985030 889 tablet_service.cc:1468] Processing CreateTablet for tablet 871b603cb92249c6b0f16c825bf46232 (DEFAULT_TABLE table=default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.985136 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bb060cdd9a48688967989d7491cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.985334 1163 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:15.985364 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871b603cb92249c6b0f16c825bf46232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.985509 1163 ts_tablet_manager.cc:1397] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.016s
I20250625 07:52:15.986527 1163 raft_consensus.cc:357] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.986742 1163 raft_consensus.cc:383] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:15.986814 1163 raft_consensus.cc:738] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:15.987030 1163 consensus_queue.cc:260] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.987716 1163 ts_tablet_manager.cc:1428] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:15.988121 1163 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:15.989111 895 tablet_service.cc:1468] Processing CreateTablet for tablet 019959e9abed46fbbbb4755cb01cef44 (DEFAULT_TABLE table=default.Table_c928e3f5ce01487c976a35b088cbe9ad [id=ea62e41d8d904cf5ac63b9d0a62cde09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.989598 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019959e9abed46fbbbb4755cb01cef44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.990516 1311 raft_consensus.cc:491] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:15.991010 1311 raft_consensus.cc:513] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:15.991663 1311 leader_election.cc:290] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:15.992877 880 tablet_service.cc:1468] Processing CreateTablet for tablet 28e490ba5d08498f921b50f057b88369 (DEFAULT_TABLE table=default.Table_0d054b4221a5407c868cac2f0d7cc131 [id=e2fb87b6822f4c9496ed3b0c5c796c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.994066 885 tablet_service.cc:1468] Processing CreateTablet for tablet ccddd630a82341cba7a857e67a349fe0 (DEFAULT_TABLE table=default.Table_4de62938b87249fdb2950c71864eeb4c [id=3a15e8259a5f4b1283f0c106d55baff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:15.994462 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e490ba5d08498f921b50f057b88369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.995931 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:15.992036 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2f72b227e14e46983c906520c29fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.999614 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccddd630a82341cba7a857e67a349fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:15.992332 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:16.002951 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.003062 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_71a35f9021c7495ca3d322bf4adf02d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.003705 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71a35f9021c7495ca3d322bf4adf02d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.003715 1025 tablet_service.cc:1468] Processing CreateTablet for tablet b2bb060cdd9a48688967989d7491cfc1 (DEFAULT_TABLE table=default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.004290 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2bb060cdd9a48688967989d7491cfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.007277 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_187c1ac9eb3649369d0c23cfd8eb18e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.007840 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.010126 1158 tablet_bootstrap.cc:492] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:15.891340 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 172) took 2108 ms. Trace:
I20250625 07:52:16.011492 1161 rpcz_store.cc:276] 0625 07:52:13.782619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.782801 (+ 182us) service_pool.cc:225] Handling call
0625 07:52:15.891185 (+2108384us) ts_tablet_manager.cc:1145] Deleting tablet 0024618293d74f559b04258f66616044
0625 07:52:15.891197 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:15.891309 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108167}
I20250625 07:52:16.011920 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 173) took 2229 ms. Trace:
I20250625 07:52:16.012030 1161 rpcz_store.cc:276] 0625 07:52:13.787585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.787699 (+ 114us) service_pool.cc:225] Handling call
0625 07:52:16.011760 (+2224061us) ts_tablet_manager.cc:1145] Deleting tablet 6b3488a9fb19452dac3241170dd2fee2
0625 07:52:16.011770 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.011880 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2217679}
I20250625 07:52:16.012369 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 174) took 2223 ms. Trace:
I20250625 07:52:16.012466 1161 rpcz_store.cc:276] 0625 07:52:13.788821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.788896 (+ 75us) service_pool.cc:225] Handling call
0625 07:52:16.012242 (+2223346us) ts_tablet_manager.cc:1145] Deleting tablet 7d9d66546d7c42949dfe5db77f1057fe
0625 07:52:16.012250 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.012331 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2210881}
I20250625 07:52:16.013360 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 871b603cb92249c6b0f16c825bf46232 (DEFAULT_TABLE table=default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.014021 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 019959e9abed46fbbbb4755cb01cef44 (DEFAULT_TABLE table=default.Table_c928e3f5ce01487c976a35b088cbe9ad [id=ea62e41d8d904cf5ac63b9d0a62cde09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.014552 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4167282133f94bfb9870bd08fc6594b4 (DEFAULT_TABLE table=default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.015043 1018 tablet_service.cc:1468] Processing CreateTablet for tablet ccddd630a82341cba7a857e67a349fe0 (DEFAULT_TABLE table=default.Table_4de62938b87249fdb2950c71864eeb4c [id=3a15e8259a5f4b1283f0c106d55baff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.016422 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 28e490ba5d08498f921b50f057b88369 (DEFAULT_TABLE table=default.Table_0d054b4221a5407c868cac2f0d7cc131 [id=e2fb87b6822f4c9496ed3b0c5c796c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.017094 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 4a2f72b227e14e46983c906520c29fa3 (DEFAULT_TABLE table=default.Table_51623e42db554f96a29174eb61aa970c [id=2bdf3240cc0946c5a47a7232d5a200db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.017477 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c81396ee6ed7400baa30dcc5d71ba45d (table default.Table_3aec9b6017944d5f8293424556b380d3 [id=975c0e95149a474b8e98ad4c4411b385]) successfully deleted
W20250625 07:52:16.017721 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0f566cc9cdd447798a5cb408ef9cd35a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f566cc9cdd447798a5cb408ef9cd35a already in progress: creating tablet
I20250625 07:52:16.018014 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3e14b60ee7a843e5bb878148d8fd2e53 (table default.Table_45646a782da0472da9dc6fca099117e6 [id=2a245f37f9f1408ab215b67ffa30ed01]) successfully deleted
I20250625 07:52:16.019052 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871b603cb92249c6b0f16c825bf46232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.010340 1158 ts_tablet_manager.cc:1397] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.158s user 0.015s sys 0.023s
I20250625 07:52:16.020432 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3c8c317c10e14332a3b3919808e4a512
I20250625 07:52:16.020924 1158 raft_consensus.cc:357] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.021134 1158 raft_consensus.cc:383] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.021262 1158 raft_consensus.cc:738] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.021500 1158 consensus_queue.cc:260] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.022092 1158 ts_tablet_manager.cc:1428] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:16.022436 1158 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:15.992410 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4167282133f94bfb9870bd08fc6594b4 (DEFAULT_TABLE table=default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.027501 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4167282133f94bfb9870bd08fc6594b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.029002 1158 tablet_bootstrap.cc:654] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.031322 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 365ef91e7bac43e283602d3d036853ff
I20250625 07:52:15.996001 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:15.996280 746 tablet_service.cc:1468] Processing CreateTablet for tablet 28e490ba5d08498f921b50f057b88369 (DEFAULT_TABLE table=default.Table_0d054b4221a5407c868cac2f0d7cc131 [id=e2fb87b6822f4c9496ed3b0c5c796c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.033506 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e490ba5d08498f921b50f057b88369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.033524 1166 tablet_replica.cc:331] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.033721 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.033983 1166 raft_consensus.cc:2241] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:16.034058 850 leader_election.cc:343] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.036058 1166 raft_consensus.cc:2270] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.038203 1166 ts_tablet_manager.cc:1905] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:16.038331 851 leader_election.cc:343] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.042692 851 leader_election.cc:304] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.043774 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4b4627f4da6b4542aff1e44157087b14 (DEFAULT_TABLE table=default.Table_7bdec7af96c94b47a5b5c890625f02cc [id=618d684f882f425dab0d9a7395deb004]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.044370 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4627f4da6b4542aff1e44157087b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.044736 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 450727d8f33e4eb489969b7974cdeeb1
I20250625 07:52:16.045356 1027 tablet_service.cc:1468] Processing CreateTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 (DEFAULT_TABLE table=default.Table_ee01c47b2df2467eadab63a9cbfb4b6a [id=860ff357dc3a4a2dad9d850c113cf464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.045965 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50c35cac8e94e088cdacb664b8adfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.046216 1311 raft_consensus.cc:2747] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.044744 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a777f64425384be5bccc86b197626618
I20250625 07:52:16.046418 1308 raft_consensus.cc:491] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.046829 594 catalog_manager.cc:6652] ReplaceTablet: tablet a777f64425384be5bccc86b197626618 deleted and replaced by tablet 6e5a207d515449e298936d8bb532437a
I20250625 07:52:16.047221 1308 raft_consensus.cc:513] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.047916 1308 leader_election.cc:290] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.048079 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4384bc544af149338514d40adf683f34
I20250625 07:52:16.049029 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32bd7cdef96946e88683e457c2f26c9b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:16.054198 850 leader_election.cc:343] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.054258 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_098cd87d74e84fe7bd51f4b61ee6ad60" } new_table_name: "default.Table_e57f684466224dd5a4a8ff79408e1a4d" modify_external_catalogs: true
I20250625 07:52:16.057575 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f3e586b96cbb4685a99ad12be406febe" } new_table_name: "default.Table_c632d0f66e464da088ba6a64e1d70e3b" modify_external_catalogs: true
I20250625 07:52:16.062711 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_26efe610b5ee4ac1b55ac4d7d91c7040"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.064119 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:16.070778 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 consensus state has no leader
W20250625 07:52:16.071471 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edda0c3f60f64ed0b965eca52c5b3591 consensus state has no leader
I20250625 07:52:16.073495 1160 ts_tablet_manager.cc:1918] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.073657 1160 log.cc:1199] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1eab049ad7fc4b5581e5c31a14491306
I20250625 07:52:16.074451 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.074576 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019959e9abed46fbbbb4755cb01cef44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.074730 1163 tablet_bootstrap.cc:654] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.075130 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccddd630a82341cba7a857e67a349fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.075357 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4167282133f94bfb9870bd08fc6594b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.075605 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e490ba5d08498f921b50f057b88369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.075884 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a2f72b227e14e46983c906520c29fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.077262 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 365ef91e7bac43e283602d3d036853ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.077601 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.078007 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4b4627f4da6b4542aff1e44157087b14 (DEFAULT_TABLE table=default.Table_7bdec7af96c94b47a5b5c890625f02cc [id=618d684f882f425dab0d9a7395deb004]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.078403 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4627f4da6b4542aff1e44157087b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.079593 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 176) took 2269 ms. Trace:
I20250625 07:52:16.079526 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32bd7cdef96946e88683e457c2f26c9b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:16.079697 1161 rpcz_store.cc:276] 0625 07:52:13.806961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.807276 (+ 315us) service_pool.cc:225] Handling call
0625 07:52:16.012653 (+2205377us) ts_tablet_manager.cc:1145] Deleting tablet 81976db0a2f94128a89842a12f067231
0625 07:52:16.012660 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.076482 (+ 63822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2180221}
I20250625 07:52:16.080098 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 175) took 2274 ms. Trace:
I20250625 07:52:16.080216 1161 rpcz_store.cc:276] 0625 07:52:13.805351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.805467 (+ 116us) service_pool.cc:225] Handling call
0625 07:52:16.079954 (+2274487us) ts_tablet_manager.cc:1145] Deleting tablet ebb943df93284913b6a520e099399322
0625 07:52:16.079962 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.080054 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2218005}
I20250625 07:52:16.080224 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4c0fff3cc3fe4f31a88c71012c45b906"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.080549 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 180) took 2188 ms. Trace:
I20250625 07:52:16.080660 1161 rpcz_store.cc:276] 0625 07:52:13.891604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:13.891937 (+ 333us) service_pool.cc:225] Handling call
0625 07:52:16.080424 (+2188487us) ts_tablet_manager.cc:1145] Deleting tablet aec412a15b964060a24631e7ebcf1ef7
0625 07:52:16.080431 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.080505 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2140424}
W20250625 07:52:16.081526 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c0fff3cc3fe4f31a88c71012c45b906 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.082859 1268 hybrid_clock.cc:648] HybridClock initialized: now 1750837936082690 us; error 167 us; skew 500 ppm
I20250625 07:52:16.083400 1268 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250625 07:52:16.088008 851 leader_election.cc:343] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.088161 851 leader_election.cc:304] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.088271 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 187) took 2066 ms. Trace:
I20250625 07:52:16.088472 1161 rpcz_store.cc:276] 0625 07:52:14.021944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.032076 (+ 10132us) service_pool.cc:225] Handling call
0625 07:52:16.080855 (+2048779us) ts_tablet_manager.cc:1145] Deleting tablet 6b1eaf0266c04873b185c1c0c2da201c
0625 07:52:16.088095 (+ 7240us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.088233 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035060}
I20250625 07:52:16.088894 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.089030 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 192) took 1959 ms. Trace:
I20250625 07:52:16.089216 1161 rpcz_store.cc:276] 0625 07:52:14.129995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.130075 (+ 80us) service_pool.cc:225] Handling call
0625 07:52:16.088735 (+1958660us) ts_tablet_manager.cc:1145] Deleting tablet 0452e73865964f709310fd22ac39a5f0
0625 07:52:16.088885 (+ 150us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.088993 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945916}
I20250625 07:52:16.090615 1160 ts_tablet_manager.cc:1939] T 1eab049ad7fc4b5581e5c31a14491306 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.090643 1308 raft_consensus.cc:2747] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.090762 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a28f5f4927f34e03b2ea4784b65cfe43" } modify_external_catalogs: true
I20250625 07:52:16.091017 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a28f5f4927f34e03b2ea4784b65cfe43" } modify_external_catalogs: true
I20250625 07:52:16.093233 882 tablet_service.cc:1468] Processing CreateTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (DEFAULT_TABLE table=default.Table_098cd87d74e84fe7bd51f4b61ee6ad60 [id=3dcf0706033c48ceaa1fdd5763284bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.094331 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.100431 1166 ts_tablet_manager.cc:1918] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.100603 1166 log.cc:1199] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/09b3830bd506443ab6f06ddd3c6b9974
I20250625 07:52:16.101045 1166 ts_tablet_manager.cc:1939] T 09b3830bd506443ab6f06ddd3c6b9974 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:16.101197 1308 raft_consensus.cc:491] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.101358 1308 raft_consensus.cc:513] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.102056 1308 leader_election.cc:290] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.103407 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e8d65c2ed5240fc923a7c7b59488e6a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:16.103502 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d5f29fa855904b50bfd2bee19de5c920" } new_table_name: "default.Table_4f95753979a34ea994409ba5978b7186" modify_external_catalogs: true
W20250625 07:52:16.103941 851 leader_election.cc:343] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.107754 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 09b3830bd506443ab6f06ddd3c6b9974 (table default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]) successfully deleted
I20250625 07:52:16.108570 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aac2883a6c934d2aa9de60cf6f345fad" } new_table_name: "default.Table_42dfb73bc9934bd293d7f1cff6e1e30f" modify_external_catalogs: true
I20250625 07:52:16.117899 1162 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:16.120913 1162 ts_tablet_manager.cc:1397] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.188s user 0.004s sys 0.000s
I20250625 07:52:16.121783 1162 raft_consensus.cc:357] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:16.121876 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0f566cc9cdd447798a5cb408ef9cd35a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f566cc9cdd447798a5cb408ef9cd35a already in progress: creating tablet
I20250625 07:52:16.121999 1162 raft_consensus.cc:383] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.122092 1162 raft_consensus.cc:738] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.122356 1162 consensus_queue.cc:260] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.122992 1162 ts_tablet_manager.cc:1428] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:16.124536 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_635498945f814a89b120fc8a6d11e3d0" } new_table_name: "default.Table_97384ebbc1584955af0a6bc425084522" modify_external_catalogs: true
I20250625 07:52:16.125083 1163 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.125298 1163 ts_tablet_manager.cc:1397] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.137s user 0.005s sys 0.000s
I20250625 07:52:16.126160 1162 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:16.127166 592 master_service.cc:948] ReplaceTablet: received request to replace tablet c65f257ae653488a8b49b0b523d2cf82 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.128125 1163 raft_consensus.cc:357] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.128448 1163 raft_consensus.cc:383] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.128719 1163 raft_consensus.cc:738] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.129191 1163 consensus_queue.cc:260] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.129799 1162 tablet_bootstrap.cc:654] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:16.130267 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3c8c317c10e14332a3b3919808e4a512 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c8c317c10e14332a3b3919808e4a512 already in progress: creating tablet
W20250625 07:52:16.130545 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4384bc544af149338514d40adf683f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4384bc544af149338514d40adf683f34 already in progress: creating tablet
W20250625 07:52:16.130738 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 365ef91e7bac43e283602d3d036853ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 365ef91e7bac43e283602d3d036853ff already in progress: creating tablet
I20250625 07:52:16.132942 1163 ts_tablet_manager.cc:1428] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20250625 07:52:16.133808 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a777f64425384be5bccc86b197626618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a777f64425384be5bccc86b197626618 already in progress: creating tablet
W20250625 07:52:16.134058 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 450727d8f33e4eb489969b7974cdeeb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 450727d8f33e4eb489969b7974cdeeb1 already in progress: creating tablet
I20250625 07:52:16.134310 1163 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.136832 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b674a39d7f9f44708b1369a5affecc82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.137557 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b674a39d7f9f44708b1369a5affecc82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.096015 1013 tablet_service.cc:1468] Processing CreateTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (DEFAULT_TABLE table=default.Table_098cd87d74e84fe7bd51f4b61ee6ad60 [id=3dcf0706033c48ceaa1fdd5763284bf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.116463 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e8d65c2ed5240fc923a7c7b59488e6a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:16.089558 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 191) took 1962 ms. Trace:
I20250625 07:52:16.143764 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.144362 1166 tablet_replica.cc:331] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.144582 1166 raft_consensus.cc:2241] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.144809 1166 raft_consensus.cc:2270] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.145774 1166 ts_tablet_manager.cc:1905] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.148707 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.149886 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1405e55703324bc7a1e26fe2c349bda7 (DEFAULT_TABLE table=default.Table_1a536939b2ae4f5b9dc8b8bc27da686f [id=bde8f673978c457e8a36b1173b315a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.150338 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1405e55703324bc7a1e26fe2c349bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.153084 1163 tablet_bootstrap.cc:654] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.153636 1166 ts_tablet_manager.cc:1918] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.153812 1166 log.cc:1199] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/012df7beedb1446ca5e533d6624afc10
I20250625 07:52:16.154230 1166 ts_tablet_manager.cc:1939] T 012df7beedb1446ca5e533d6624afc10 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:16.155035 850 leader_election.cc:343] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.155184 850 leader_election.cc:304] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.155617 1308 raft_consensus.cc:2747] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.157403 1315 raft_consensus.cc:491] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.159525 746 tablet_service.cc:1468] Processing CreateTablet for tablet 547b758c2e6c4558bb232f14b37aa81a (DEFAULT_TABLE table=default.Table_24edc711d8fd46d7b8d3e7d20be99016 [id=83522331e5df4bafaa65c81be08c7525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.160049 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 547b758c2e6c4558bb232f14b37aa81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.163800 1315 raft_consensus.cc:513] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.164965 750 tablet_service.cc:1468] Processing CreateTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 (DEFAULT_TABLE table=default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.165341 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe5c37fc9bd49c685de8f469a7af0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.166281 1315 leader_election.cc:290] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:16.171768 986 leader_election.cc:343] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:16.172752 1308 raft_consensus.cc:491] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.172950 1308 raft_consensus.cc:513] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.173641 1308 leader_election.cc:290] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.176085 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 012df7beedb1446ca5e533d6624afc10 (table default.Table_978181531aa34778bd274293434b1ac4 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:16.176045 1161 rpcz_store.cc:276] 0625 07:52:14.126876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.127865 (+ 989us) service_pool.cc:225] Handling call
0625 07:52:16.089444 (+1961579us) ts_tablet_manager.cc:1145] Deleting tablet 09b3830bd506443ab6f06ddd3c6b9974
0625 07:52:16.089454 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.089528 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940584}
I20250625 07:52:16.177733 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:16.178911 851 leader_election.cc:343] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:16.183049 985 leader_election.cc:343] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:16.183312 985 leader_election.cc:304] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:16.183645 1315 raft_consensus.cc:2747] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:16.189019 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32bd7cdef96946e88683e457c2f26c9b consensus state has no leader
W20250625 07:52:16.190366 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06e4bf78218f4b82b4f16f59b1704256 consensus state has no leader
W20250625 07:52:16.190924 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29eaa8d93e5b4d1d8ed147b60c7da859 consensus state has no leader
I20250625 07:52:16.197435 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 31942f0998bb4118a91f476325851d5c
I20250625 07:52:16.198904 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_efc0d4ed62e24872b76dc943522353cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.208189 1308 raft_consensus.cc:491] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:16.208389 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc0d4ed62e24872b76dc943522353cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.208388 1308 raft_consensus.cc:513] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.209268 1308 leader_election.cc:290] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.210041 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:16.210027 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:16.210791 851 leader_election.cc:343] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.212771 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_56d1c8028ad54c7d8dcb1f2b342f9607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.214751 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d1c8028ad54c7d8dcb1f2b342f9607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:16.218742 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 31942f0998bb4118a91f476325851d5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31942f0998bb4118a91f476325851d5c already in progress: creating tablet
I20250625 07:52:16.222296 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dbf9b53cf6f843c1b02c9793e9bc0741" } modify_external_catalogs: true
I20250625 07:52:16.222478 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dbf9b53cf6f843c1b02c9793e9bc0741" } modify_external_catalogs: true
I20250625 07:52:16.223207 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5a58b057808640f58c4b2244c66f56bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.223718 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a58b057808640f58c4b2244c66f56bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.229434 1308 raft_consensus.cc:491] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.229640 1308 raft_consensus.cc:513] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.230345 1308 leader_election.cc:290] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.231858 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec412a15b964060a24631e7ebcf1ef7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:16.232512 851 leader_election.cc:343] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.238660 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aacce885498f4090b6087bc73c1ba997" } modify_external_catalogs: true
I20250625 07:52:16.238862 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aacce885498f4090b6087bc73c1ba997" } modify_external_catalogs: true
I20250625 07:52:16.242983 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.243790 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 365ef91e7bac43e283602d3d036853ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.244043 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.244374 884 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.244437 1160 tablet_replica.cc:331] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.244681 1160 raft_consensus.cc:2241] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.244891 1160 raft_consensus.cc:2270] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.245066 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.245496 1160 ts_tablet_manager.cc:1905] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.245819 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1405e55703324bc7a1e26fe2c349bda7 (DEFAULT_TABLE table=default.Table_1a536939b2ae4f5b9dc8b8bc27da686f [id=bde8f673978c457e8a36b1173b315a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.246285 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1405e55703324bc7a1e26fe2c349bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.250016 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0ba3665554d4b36a261ce71b5fd743c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.250625 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0ba3665554d4b36a261ce71b5fd743c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.252127 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:16.252749 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0bee6f682a14fa59118497e9a616ebf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.253278 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0bee6f682a14fa59118497e9a616ebf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:16.253482 850 leader_election.cc:343] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.253607 850 leader_election.cc:304] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.254367 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 6239713a586d43c28c101ba2447d579d from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.254439 1308 raft_consensus.cc:2747] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.254935 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec412a15b964060a24631e7ebcf1ef7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:16.255167 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:16.255599 850 leader_election.cc:343] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.255719 850 leader_election.cc:304] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:16.256186 850 leader_election.cc:343] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.256304 850 leader_election.cc:304] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.256785 1311 raft_consensus.cc:2747] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.256973 1311 raft_consensus.cc:2747] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.257310 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2908cac0353642cc92bc2152162de70e" } modify_external_catalogs: true
I20250625 07:52:16.257472 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2908cac0353642cc92bc2152162de70e" } modify_external_catalogs: true
I20250625 07:52:16.257568 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.257961 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c1cbfecab414abaa74c66f1773f817e" } modify_external_catalogs: true
I20250625 07:52:16.258143 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c1cbfecab414abaa74c66f1773f817e" } modify_external_catalogs: true
I20250625 07:52:16.258649 884 tablet_service.cc:1468] Processing CreateTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 (DEFAULT_TABLE table=default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.259109 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe5c37fc9bd49c685de8f469a7af0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.259359 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_85d42dece3ab4be5a22696c28417f194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.259868 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85d42dece3ab4be5a22696c28417f194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.261322 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.262966 882 tablet_service.cc:1468] Processing CreateTablet for tablet 547b758c2e6c4558bb232f14b37aa81a (DEFAULT_TABLE table=default.Table_24edc711d8fd46d7b8d3e7d20be99016 [id=83522331e5df4bafaa65c81be08c7525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.263358 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 547b758c2e6c4558bb232f14b37aa81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.264308 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 365ef91e7bac43e283602d3d036853ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.264469 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 193) took 2109 ms. Trace:
I20250625 07:52:16.264731 1161 rpcz_store.cc:276] 0625 07:52:14.142579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.154473 (+ 11894us) service_pool.cc:225] Handling call
0625 07:52:16.176430 (+2021957us) ts_tablet_manager.cc:1145] Deleting tablet 21658216da3544a2b8b8f39667782ffb
0625 07:52:16.176440 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.251875 (+ 75435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993639}
I20250625 07:52:16.265225 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 197) took 2066 ms. Trace:
I20250625 07:52:16.265544 1161 rpcz_store.cc:276] 0625 07:52:14.198408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.199225 (+ 817us) service_pool.cc:225] Handling call
0625 07:52:16.265085 (+2065860us) ts_tablet_manager.cc:1145] Deleting tablet b6382963863548398bed90f03b840806
0625 07:52:16.265094 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.265187 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062707}
I20250625 07:52:16.266096 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 198) took 2066 ms. Trace:
I20250625 07:52:16.266142 1160 ts_tablet_manager.cc:1918] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.266387 1161 rpcz_store.cc:276] 0625 07:52:14.203179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.203247 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:16.265761 (+2062514us) ts_tablet_manager.cc:1145] Deleting tablet 4c3c180785b64f7d85863c14cfd09ccc
0625 07:52:16.265769 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.266063 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2058728}
I20250625 07:52:16.266945 1160 log.cc:1199] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/365ef91e7bac43e283602d3d036853ff
I20250625 07:52:16.267256 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 201) took 1971 ms. Trace:
I20250625 07:52:16.267489 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.267560 1160 ts_tablet_manager.cc:1939] T 365ef91e7bac43e283602d3d036853ff P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.267676 1161 rpcz_store.cc:276] 0625 07:52:14.295454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.295498 (+ 44us) service_pool.cc:225] Handling call
0625 07:52:16.266596 (+1971098us) ts_tablet_manager.cc:1145] Deleting tablet b377c8111c9f4ac2ba05e65694d611b9
0625 07:52:16.266606 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.266694 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970891}
I20250625 07:52:16.268047 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 202) took 1972 ms. Trace:
I20250625 07:52:16.268596 1161 rpcz_store.cc:276] 0625 07:52:14.296015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.296065 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:16.267927 (+1971862us) ts_tablet_manager.cc:1145] Deleting tablet c65f257ae653488a8b49b0b523d2cf82
0625 07:52:16.267938 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.268019 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1971640}
I20250625 07:52:16.269173 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 200) took 1974 ms. Trace:
I20250625 07:52:16.270066 1161 rpcz_store.cc:276] 0625 07:52:14.295121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.298499 (+ 3378us) service_pool.cc:225] Handling call
0625 07:52:16.268935 (+1970436us) ts_tablet_manager.cc:1145] Deleting tablet c7a2b94af59648239cfd1e5ce5e8b968
0625 07:52:16.268944 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.269144 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970234}
I20250625 07:52:16.277400 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.278069 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_9babc704c99d4319bf7f3906ef371e72 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 1 got 0
I20250625 07:52:16.279479 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.279966 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_75ee0958612248248c0bfe106c759571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.280540 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75ee0958612248248c0bfe106c759571 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.281258 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_978181531aa34778bd274293434b1ac4" } new_table_name: "default.Table_14bfc8ca6e024a2bba684efdcfc28fd3" modify_external_catalogs: true
I20250625 07:52:16.281764 1024 tablet_service.cc:1468] Processing CreateTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 (DEFAULT_TABLE table=default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.282164 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe5c37fc9bd49c685de8f469a7af0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.283334 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.284729 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1405e55703324bc7a1e26fe2c349bda7 (DEFAULT_TABLE table=default.Table_1a536939b2ae4f5b9dc8b8bc27da686f [id=bde8f673978c457e8a36b1173b315a06]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.287983 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0024618293d74f559b04258f66616044 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0024618293d74f559b04258f66616044 already in progress: creating tablet
W20250625 07:52:16.289026 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6b3488a9fb19452dac3241170dd2fee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b3488a9fb19452dac3241170dd2fee2 already in progress: creating tablet
I20250625 07:52:16.289428 1158 tablet_bootstrap.cc:492] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
W20250625 07:52:16.289753 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7d9d66546d7c42949dfe5db77f1057fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d9d66546d7c42949dfe5db77f1057fe already in progress: creating tablet
W20250625 07:52:16.290583 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 81976db0a2f94128a89842a12f067231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81976db0a2f94128a89842a12f067231 already in progress: creating tablet
W20250625 07:52:16.290871 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ebb943df93284913b6a520e099399322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebb943df93284913b6a520e099399322 already in progress: creating tablet
W20250625 07:52:16.291168 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet aec412a15b964060a24631e7ebcf1ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aec412a15b964060a24631e7ebcf1ef7 already in progress: creating tablet
W20250625 07:52:16.291452 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0452e73865964f709310fd22ac39a5f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0452e73865964f709310fd22ac39a5f0 already in progress: creating tablet
W20250625 07:52:16.292037 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 09b3830bd506443ab6f06ddd3c6b9974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09b3830bd506443ab6f06ddd3c6b9974 already in progress: creating tablet
W20250625 07:52:16.292755 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 21658216da3544a2b8b8f39667782ffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21658216da3544a2b8b8f39667782ffb already in progress: creating tablet
I20250625 07:52:16.292805 1158 ts_tablet_manager.cc:1397] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.268s user 0.020s sys 0.055s
W20250625 07:52:16.293093 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b6382963863548398bed90f03b840806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6382963863548398bed90f03b840806 already in progress: creating tablet
W20250625 07:52:16.293380 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4c3c180785b64f7d85863c14cfd09ccc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c3c180785b64f7d85863c14cfd09ccc already in progress: creating tablet
W20250625 07:52:16.293638 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b377c8111c9f4ac2ba05e65694d611b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b377c8111c9f4ac2ba05e65694d611b9 already in progress: creating tablet
I20250625 07:52:16.293632 1158 raft_consensus.cc:357] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:16.293977 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c65f257ae653488a8b49b0b523d2cf82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65f257ae653488a8b49b0b523d2cf82 already in progress: creating tablet
I20250625 07:52:16.294188 1158 raft_consensus.cc:383] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:16.294225 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c7a2b94af59648239cfd1e5ce5e8b968 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a2b94af59648239cfd1e5ce5e8b968 already in progress: creating tablet
I20250625 07:52:16.294301 1158 raft_consensus.cc:738] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.294602 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1405e55703324bc7a1e26fe2c349bda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.295131 1158 consensus_queue.cc:260] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.295734 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1eab049ad7fc4b5581e5c31a14491306 (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
I20250625 07:52:16.296958 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 365ef91e7bac43e283602d3d036853ff (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
I20250625 07:52:16.297158 750 tablet_service.cc:1468] Processing CreateTablet for tablet f4bb74391a1c4e2792226d259a07f977 (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.297546 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 450727d8f33e4eb489969b7974cdeeb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 450727d8f33e4eb489969b7974cdeeb1 already in progress: creating tablet
W20250625 07:52:16.297825 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a777f64425384be5bccc86b197626618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a777f64425384be5bccc86b197626618 already in progress: creating tablet
I20250625 07:52:16.297808 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 547b758c2e6c4558bb232f14b37aa81a (DEFAULT_TABLE table=default.Table_24edc711d8fd46d7b8d3e7d20be99016 [id=83522331e5df4bafaa65c81be08c7525]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.298141 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4384bc544af149338514d40adf683f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4384bc544af149338514d40adf683f34 already in progress: creating tablet
W20250625 07:52:16.298419 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3c8c317c10e14332a3b3919808e4a512 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c8c317c10e14332a3b3919808e4a512 already in progress: creating tablet
I20250625 07:52:16.298904 745 tablet_service.cc:1468] Processing CreateTablet for tablet faf049714a7e4d7eb947c29f272bbb7c (DEFAULT_TABLE table=default.Table_71a35f9021c7495ca3d322bf4adf02d6 [id=a1e6b3ecd3bb49ef94da294bb866f824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.300093 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 203) took 1751 ms. Trace:
I20250625 07:52:16.300221 1161 rpcz_store.cc:276] 0625 07:52:14.524938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.526282 (+ 1344us) service_pool.cc:225] Handling call
0625 07:52:16.276478 (+1750196us) ts_tablet_manager.cc:1145] Deleting tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe
0625 07:52:16.276489 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.276816 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749978}
I20250625 07:52:16.300529 1161 tablet_replica.cc:331] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:16.300592 744 tablet_service.cc:1468] Processing CreateTablet for tablet e109c9dca7444681ac517c7fc3e14dfa (DEFAULT_TABLE table=default.Table_4c0fff3cc3fe4f31a88c71012c45b906 [id=eb23c86687b2498e8b2bbb0f3bb37db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.301031 1161 raft_consensus.cc:2241] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.301198 1161 raft_consensus.cc:2270] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.301514 747 tablet_service.cc:1468] Processing CreateTablet for tablet 110fe299c7c44df69f459da72a1138a5 (DEFAULT_TABLE table=default.Table_a48b063f27e347c2974134be14e26739 [id=d281de0430994f13a49c3d53c062dec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.301738 1161 ts_tablet_manager.cc:1905] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.301779 749 tablet_service.cc:1468] Processing CreateTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 (DEFAULT_TABLE table=default.Table_15745c1b68f54870b34b4ced0001369f [id=2cef554f33094c77b8684986968dadef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.301859 1163 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.302021 1163 ts_tablet_manager.cc:1397] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.168s user 0.004s sys 0.000s
I20250625 07:52:16.302102 754 tablet_service.cc:1468] Processing CreateTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c (DEFAULT_TABLE table=default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.302742 743 tablet_service.cc:1468] Processing CreateTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 (DEFAULT_TABLE table=default.Table_817356d9efcf4f889897f9e942387859 [id=989d9b62801c467e8bba2e03c0f13395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.302860 1163 raft_consensus.cc:357] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.303047 1163 raft_consensus.cc:383] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.303108 1163 raft_consensus.cc:738] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.303202 757 tablet_service.cc:1468] Processing CreateTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 (DEFAULT_TABLE table=default.Table_3200657505f94bf7a15245040bb3b059 [id=c80e78f1ef124af2b8f405e7d162ade8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.303359 1163 consensus_queue.cc:260] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.303880 1163 ts_tablet_manager.cc:1428] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:16.304225 1163 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.305562 758 tablet_service.cc:1468] Processing CreateTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.305891 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6e5a207d515449e298936d8bb532437a (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.309288 1163 tablet_bootstrap.cc:654] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.310245 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 547b758c2e6c4558bb232f14b37aa81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.312294 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:16.313081 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_42dfb73bc9934bd293d7f1cff6e1e30f" } new_table_name: "default.Table_af477e9a0d964469882fdef9caf4ca71" modify_external_catalogs: true
I20250625 07:52:16.313406 1158 ts_tablet_manager.cc:1428] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250625 07:52:16.313907 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:16.315060 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4fdb6a689ce6440ab7037484e2b9cdc5
W20250625 07:52:16.315732 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10c7f4b4187c428bbec136c85219da4d consensus state has no leader
I20250625 07:52:16.315788 1162 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:16.315971 1162 ts_tablet_manager.cc:1397] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.190s user 0.032s sys 0.038s
I20250625 07:52:16.316171 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6239713a586d43c28c101ba2447d579d
I20250625 07:52:16.316558 595 catalog_manager.cc:6652] ReplaceTablet: tablet 6239713a586d43c28c101ba2447d579d deleted and replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:16.316885 1162 raft_consensus.cc:357] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.317135 1162 raft_consensus.cc:383] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.317247 1162 raft_consensus.cc:738] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.317560 1162 consensus_queue.cc:260] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.318815 1162 ts_tablet_manager.cc:1428] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:16.319154 1162 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:16.320475 1163 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.320709 1163 ts_tablet_manager.cc:1397] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250625 07:52:16.321561 1163 raft_consensus.cc:357] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.321857 1163 raft_consensus.cc:383] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.322022 1163 raft_consensus.cc:738] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.322346 1163 consensus_queue.cc:260] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.322983 1163 ts_tablet_manager.cc:1428] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:16.323894 1163 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.323421 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bb74391a1c4e2792226d259a07f977. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.325949 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a338005a7b44b9b0fba7a7096cfcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.326165 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110fe299c7c44df69f459da72a1138a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.327992 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6e5a207d515449e298936d8bb532437a
I20250625 07:52:16.328861 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf049714a7e4d7eb947c29f272bbb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.330266 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 1222f2c1027b42c6ad9627359a93d678 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fdb6a689ce6440ab7037484e2b9cdc5) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.331365 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d393a85f5fde4b3fb57e5b3ec372791c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.332183 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c9321a70e44b5bb3ea419bb466fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.336300 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 86ea9df068a94d9187f2f9620e371a39 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.338400 1163 tablet_bootstrap.cc:654] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.339814 1158 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:16.341784 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04016cda7f746d49e30be3b1c9c18b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:16.342141 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 31942f0998bb4118a91f476325851d5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31942f0998bb4118a91f476325851d5c already in progress: creating tablet
I20250625 07:52:16.342582 1268 webserver.cc:469] Webserver started at http://127.0.104.62:39255/ using document root <none> and password file <none>
I20250625 07:52:16.343868 1268 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:16.344034 1268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:16.345007 882 tablet_service.cc:1468] Processing CreateTablet for tablet f4bb74391a1c4e2792226d259a07f977 (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.345888 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bb74391a1c4e2792226d259a07f977. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.346323 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5a207d515449e298936d8bb532437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.347755 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abb4d9dc2da7453d8336d5589dda3a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.348647 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e109c9dca7444681ac517c7fc3e14dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.353718 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2be2bfc1127842c1aac7e83c5725e760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.355230 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be2bfc1127842c1aac7e83c5725e760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.355760 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4124f127e3e54adab2a8e666a1f18e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.356244 1163 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.356297 1158 tablet_bootstrap.cc:654] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:16.356333 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4124f127e3e54adab2a8e666a1f18e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.356405 1163 ts_tablet_manager.cc:1397] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20250625 07:52:16.357475 1163 raft_consensus.cc:357] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.357695 1163 raft_consensus.cc:383] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.357789 1163 raft_consensus.cc:738] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.358048 1163 consensus_queue.cc:260] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.358232 880 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.358670 1163 ts_tablet_manager.cc:1428] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:16.358780 880 tablet_service.cc:1468] Processing CreateTablet for tablet 110fe299c7c44df69f459da72a1138a5 (DEFAULT_TABLE table=default.Table_a48b063f27e347c2974134be14e26739 [id=d281de0430994f13a49c3d53c062dec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.358873 884 tablet_service.cc:1468] Processing CreateTablet for tablet e109c9dca7444681ac517c7fc3e14dfa (DEFAULT_TABLE table=default.Table_4c0fff3cc3fe4f31a88c71012c45b906 [id=eb23c86687b2498e8b2bbb0f3bb37db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.359006 1163 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.359294 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110fe299c7c44df69f459da72a1138a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.359508 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.359730 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.360114 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 32bd7cdef96946e88683e457c2f26c9b from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.360201 885 tablet_service.cc:1468] Processing CreateTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 (DEFAULT_TABLE table=default.Table_3200657505f94bf7a15245040bb3b059 [id=c80e78f1ef124af2b8f405e7d162ade8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.360559 881 tablet_service.cc:1468] Processing CreateTablet for tablet faf049714a7e4d7eb947c29f272bbb7c (DEFAULT_TABLE table=default.Table_71a35f9021c7495ca3d322bf4adf02d6 [id=a1e6b3ecd3bb49ef94da294bb866f824]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.361064 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe already in progress: creating tablet
I20250625 07:52:16.361280 1160 tablet_replica.cc:331] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.361544 1160 raft_consensus.cc:2241] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.362094 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.362808 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.362962 754 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.363339 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.363435 892 tablet_service.cc:1468] Processing CreateTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 (DEFAULT_TABLE table=default.Table_15745c1b68f54870b34b4ced0001369f [id=2cef554f33094c77b8684986968dadef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.363744 894 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.364030 889 tablet_service.cc:1468] Processing CreateTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 (DEFAULT_TABLE table=default.Table_817356d9efcf4f889897f9e942387859 [id=989d9b62801c467e8bba2e03c0f13395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.364146 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.364470 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.364578 887 tablet_service.cc:1468] Processing CreateTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c (DEFAULT_TABLE table=default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.364779 1160 raft_consensus.cc:2270] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.365162 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6e5a207d515449e298936d8bb532437a (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.365398 1160 ts_tablet_manager.cc:1905] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.365613 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.365727 895 tablet_service.cc:1468] Processing CreateTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.359515 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e109c9dca7444681ac517c7fc3e14dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.367304 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04016cda7f746d49e30be3b1c9c18b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.368799 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.369614 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.370509 1162 tablet_bootstrap.cc:654] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.372098 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 110fe299c7c44df69f459da72a1138a5 (DEFAULT_TABLE table=default.Table_a48b063f27e347c2974134be14e26739 [id=d281de0430994f13a49c3d53c062dec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.372241 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.372531 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110fe299c7c44df69f459da72a1138a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.376832 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d393a85f5fde4b3fb57e5b3ec372791c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.377995 1024 tablet_service.cc:1468] Processing CreateTablet for tablet faf049714a7e4d7eb947c29f272bbb7c (DEFAULT_TABLE table=default.Table_71a35f9021c7495ca3d322bf4adf02d6 [id=a1e6b3ecd3bb49ef94da294bb866f824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.378404 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf049714a7e4d7eb947c29f272bbb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.379381 1025 tablet_service.cc:1468] Processing CreateTablet for tablet e109c9dca7444681ac517c7fc3e14dfa (DEFAULT_TABLE table=default.Table_4c0fff3cc3fe4f31a88c71012c45b906 [id=eb23c86687b2498e8b2bbb0f3bb37db4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.379952 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e109c9dca7444681ac517c7fc3e14dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.380066 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.380542 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.381249 1166 tablet_replica.cc:331] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.381249 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.381438 1166 raft_consensus.cc:2241] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.381843 1166 raft_consensus.cc:2270] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.383183 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.386096 1028 tablet_service.cc:1468] Processing CreateTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 (DEFAULT_TABLE table=default.Table_3200657505f94bf7a15245040bb3b059 [id=c80e78f1ef124af2b8f405e7d162ade8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.386498 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abb4d9dc2da7453d8336d5589dda3a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.391072 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6e5a207d515449e298936d8bb532437a (DEFAULT_TABLE table=default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.391135 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e498d11f2eb04d23b580c78f5e2f3fcf" } new_table_name: "default.Table_373a265cd8044801a0fba0f411968870" modify_external_catalogs: true
I20250625 07:52:16.392637 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5a207d515449e298936d8bb532437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.383359 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faf049714a7e4d7eb947c29f272bbb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.393123 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_46671c5a6dee4493836a690b730763e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.393702 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46671c5a6dee4493836a690b730763e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.393986 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5a207d515449e298936d8bb532437a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.392459 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.396871 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abb4d9dc2da7453d8336d5589dda3a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.398392 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a338005a7b44b9b0fba7a7096cfcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.400844 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c9321a70e44b5bb3ea419bb466fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.401765 1166 ts_tablet_manager.cc:1905] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.403251 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a48b063f27e347c2974134be14e26739" } new_table_name: "default.Table_55794188bd9d47d29bc77e1c1498512f" modify_external_catalogs: true
I20250625 07:52:16.404806 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a940f52db1ba4b5caa2137bf0827a978" } modify_external_catalogs: true
I20250625 07:52:16.404975 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a940f52db1ba4b5caa2137bf0827a978" } modify_external_catalogs: true
I20250625 07:52:16.407421 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d6bdc3cfe42242a5a341dd866d9ba2ea" } new_table_name: "default.Table_e46845b5de944ab384d68468026acafc" modify_external_catalogs: true
I20250625 07:52:16.408370 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.409293 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086c9321a70e44b5bb3ea419bb466fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.410216 1315 raft_consensus.cc:491] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.410374 1315 raft_consensus.cc:513] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.411268 1315 leader_election.cc:290] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:16.412145 986 leader_election.cc:343] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:16.418452 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 (DEFAULT_TABLE table=default.Table_15745c1b68f54870b34b4ced0001369f [id=2cef554f33094c77b8684986968dadef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.419539 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a338005a7b44b9b0fba7a7096cfcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.420348 1268 fs_manager.cc:714] Time spent opening directory manager: real 0.074s user 0.005s sys 0.001s
W20250625 07:52:16.422036 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32bd7cdef96946e88683e457c2f26c9b consensus state has no leader
W20250625 07:52:16.423287 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fdb6a689ce6440ab7037484e2b9cdc5 already in progress: creating tablet
W20250625 07:52:16.423586 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet cbdbd9230210455ba615d51f1a98267b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdbd9230210455ba615d51f1a98267b already in progress: creating tablet
W20250625 07:52:16.423785 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e289ea5553ac4ff983871ece4fc72ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e289ea5553ac4ff983871ece4fc72ee5 already in progress: creating tablet
W20250625 07:52:16.423961 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6e5a207d515449e298936d8bb532437a because the tablet was not found. No further retry: Not found: Tablet not found: 6e5a207d515449e298936d8bb532437a
W20250625 07:52:16.424147 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6239713a586d43c28c101ba2447d579d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6239713a586d43c28c101ba2447d579d already in progress: creating tablet
I20250625 07:52:16.428879 1308 raft_consensus.cc:491] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.418303 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 (DEFAULT_TABLE table=default.Table_817356d9efcf4f889897f9e942387859 [id=989d9b62801c467e8bba2e03c0f13395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.433928 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04016cda7f746d49e30be3b1c9c18b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.437206 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b2a733ce9ffb4a74b2c8f1716dc57187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.437832 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_72e90a4803a04ea09a7b37b41bc7b65a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.438419 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72e90a4803a04ea09a7b37b41bc7b65a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.438618 1161 ts_tablet_manager.cc:1918] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.438776 1161 log.cc:1199] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/99fd0fdac6c24c0aad6f25e2f732c3d6
I20250625 07:52:16.439194 1161 ts_tablet_manager.cc:1939] T 99fd0fdac6c24c0aad6f25e2f732c3d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:16.440172 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2a733ce9ffb4a74b2c8f1716dc57187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.441423 1013 tablet_service.cc:1468] Processing CreateTablet for tablet f4bb74391a1c4e2792226d259a07f977 (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.442298 1366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:16.445291 1308 raft_consensus.cc:513] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.445600 1268 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.010s sys 0.006s
I20250625 07:52:16.445957 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4bb74391a1c4e2792226d259a07f977. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.445834 1268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
uuid: "00a3faeffaa4422ea73b98ee74768a4a"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:16.447194 1268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:16.447453 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 204) took 1914 ms. Trace:
I20250625 07:52:16.447604 1161 rpcz_store.cc:276] 0625 07:52:14.525207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.526532 (+ 1325us) service_pool.cc:225] Handling call
0625 07:52:16.300471 (+1773939us) ts_tablet_manager.cc:1145] Deleting tablet 99fd0fdac6c24c0aad6f25e2f732c3d6
0625 07:52:16.300480 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.435300 (+134820us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.438490 (+ 3190us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.439491 (+ 1001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773753}
I20250625 07:52:16.448175 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 205) took 1922 ms. Trace:
I20250625 07:52:16.448287 1161 rpcz_store.cc:276] 0625 07:52:14.525937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.526752 (+ 815us) service_pool.cc:225] Handling call
0625 07:52:16.447958 (+1921206us) ts_tablet_manager.cc:1145] Deleting tablet 250175f54ba94a9fbc2455d9cf27fb2e
0625 07:52:16.447968 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.448109 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921058}
I20250625 07:52:16.448654 1161 tablet_replica.cc:331] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:16.448926 1161 raft_consensus.cc:2241] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.449621 1162 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:16.449817 1162 ts_tablet_manager.cc:1397] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.005s
I20250625 07:52:16.450724 1162 raft_consensus.cc:357] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.450920 1162 raft_consensus.cc:383] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.451026 1162 raft_consensus.cc:738] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.451324 1162 consensus_queue.cc:260] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.453279 1162 ts_tablet_manager.cc:1428] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:16.453586 1162 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:16.453613 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95a763abda074d358f176fd7e7be0e5c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:16.454391 851 leader_election.cc:343] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.455432 1162 tablet_bootstrap.cc:654] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.456305 1166 ts_tablet_manager.cc:1918] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.456429 1166 log.cc:1199] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/86ea9df068a94d9187f2f9620e371a39
I20250625 07:52:16.456800 1166 ts_tablet_manager.cc:1939] T 86ea9df068a94d9187f2f9620e371a39 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:16.466212 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 110fe299c7c44df69f459da72a1138a5 consensus state has no leader
I20250625 07:52:16.466625 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2d5b724ca63c436ba1ee776b8df3cec9
W20250625 07:52:16.467252 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e41e53efb6f342d2b1e4d642f7aaad10 consensus state has no leader
I20250625 07:52:16.467626 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.469547 895 tablet_service.cc:1468] Processing CreateTablet for tablet 97b9a14151fc472e956126620633be32 (DEFAULT_TABLE table=default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.470045 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b9a14151fc472e956126620633be32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.471011 885 tablet_service.cc:1468] Processing CreateTablet for tablet cf175e17865e40ceb26aa955b91c2c13 (DEFAULT_TABLE table=default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.471413 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf175e17865e40ceb26aa955b91c2c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.472980 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfcef152c9844a59989c532bd60c07d (DEFAULT_TABLE table=default.Table_56d1c8028ad54c7d8dcb1f2b342f9607 [id=b6cfc505ed734958966b9e75b0e39415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.473505 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfcef152c9844a59989c532bd60c07d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:16.474527 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b3c54204d5446c688cb123edbd430c1 consensus state has no leader
I20250625 07:52:16.477166 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5199775108e44a9780009e4788a861ad (DEFAULT_TABLE table=default.Table_5a58b057808640f58c4b2244c66f56bf [id=9225c1ea9b174d47a1bb9421e6d251c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.477576 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 86ea9df068a94d9187f2f9620e371a39 (table default.Table_17d5751097fa40feaf90d0fb3c97e1b5 [id=5281aca609cf421990f96de3424b4442]) successfully deleted
I20250625 07:52:16.478140 747 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.478381 1166 tablet_replica.cc:331] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.478945 1166 raft_consensus.cc:2241] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.479121 1166 raft_consensus.cc:2270] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.479887 1166 ts_tablet_manager.cc:1905] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.481343 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:16.482600 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 32bd7cdef96946e88683e457c2f26c9b (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 32bd7cdef96946e88683e457c2f26c9b consensus state has no leader
I20250625 07:52:16.484180 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfcef152c9844a59989c532bd60c07d (DEFAULT_TABLE table=default.Table_56d1c8028ad54c7d8dcb1f2b342f9607 [id=b6cfc505ed734958966b9e75b0e39415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.484639 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfcef152c9844a59989c532bd60c07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.485318 744 tablet_service.cc:1468] Processing CreateTablet for tablet 5199775108e44a9780009e4788a861ad (DEFAULT_TABLE table=default.Table_5a58b057808640f58c4b2244c66f56bf [id=9225c1ea9b174d47a1bb9421e6d251c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.486883 747 tablet_service.cc:1468] Processing CreateTablet for tablet 97b9a14151fc472e956126620633be32 (DEFAULT_TABLE table=default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.487131 757 tablet_service.cc:1468] Processing CreateTablet for tablet cf175e17865e40ceb26aa955b91c2c13 (DEFAULT_TABLE table=default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.487327 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b9a14151fc472e956126620633be32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.487560 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf175e17865e40ceb26aa955b91c2c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.488463 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c (DEFAULT_TABLE table=default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.489377 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5199775108e44a9780009e4788a861ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.489351 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.490849 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.492401 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.492974 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_66abf1d5bf9b498aacef39657e91e79c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.493479 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:16.493588 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66abf1d5bf9b498aacef39657e91e79c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.494431 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d393a85f5fde4b3fb57e5b3ec372791c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.499385 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.499704 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95a763abda074d358f176fd7e7be0e5c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:16.504050 1308 leader_election.cc:290] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:16.504345 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5199775108e44a9780009e4788a861ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.504487 1160 ts_tablet_manager.cc:1918] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:16.504957 850 leader_election.cc:343] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.505108 850 leader_election.cc:304] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.505787 1308 raft_consensus.cc:2747] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.509135 1160 log.cc:1199] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/aec412a15b964060a24631e7ebcf1ef7
I20250625 07:52:16.509701 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 32bd7cdef96946e88683e457c2f26c9b
I20250625 07:52:16.509928 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 572b063d66434e4796f6b86f8221e3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.510247 588 catalog_manager.cc:6652] ReplaceTablet: tablet 32bd7cdef96946e88683e457c2f26c9b deleted and replaced by tablet 2db37ece7a964f47911fe4fc7434cd07
I20250625 07:52:16.511474 1160 ts_tablet_manager.cc:1939] T aec412a15b964060a24631e7ebcf1ef7 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.512477 1161 raft_consensus.cc:2270] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.512634 1160 tablet_replica.cc:331] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.513589 1160 raft_consensus.cc:2241] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.513775 1160 raft_consensus.cc:2270] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.514700 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet aec412a15b964060a24631e7ebcf1ef7 (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:16.516076 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98fc1c8ebdd141a3b0500bab94206f7b" } modify_external_catalogs: true
I20250625 07:52:16.516274 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98fc1c8ebdd141a3b0500bab94206f7b" } modify_external_catalogs: true
I20250625 07:52:16.517050 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2be2bfc1127842c1aac7e83c5725e760" } modify_external_catalogs: true
I20250625 07:52:16.517228 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2be2bfc1127842c1aac7e83c5725e760" } modify_external_catalogs: true
I20250625 07:52:16.517674 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5946e2f2580849578b2bda787459faef" } modify_external_catalogs: true
I20250625 07:52:16.517835 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5946e2f2580849578b2bda787459faef" } modify_external_catalogs: true
I20250625 07:52:16.518947 1166 ts_tablet_manager.cc:1918] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.519714 1160 ts_tablet_manager.cc:1905] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.519623 1166 log.cc:1199] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b7139335c7124ab79bc2d0a251984199
I20250625 07:52:16.521514 1166 ts_tablet_manager.cc:1939] T b7139335c7124ab79bc2d0a251984199 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:16.522696 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d5b724ca63c436ba1ee776b8df3cec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.522884 1315 raft_consensus.cc:491] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.524974 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 97b9a14151fc472e956126620633be32 (DEFAULT_TABLE table=default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.525369 1166 tablet_replica.cc:331] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.525529 1166 raft_consensus.cc:2241] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.525545 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 32bd7cdef96946e88683e457c2f26c9b with delete_type TABLET_DATA_DELETED (replaced by tablet 2db37ece7a964f47911fe4fc7434cd07) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.525739 1166 raft_consensus.cc:2270] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.526582 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfcef152c9844a59989c532bd60c07d (DEFAULT_TABLE table=default.Table_56d1c8028ad54c7d8dcb1f2b342f9607 [id=b6cfc505ed734958966b9e75b0e39415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.527108 1166 ts_tablet_manager.cc:1905] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.528383 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0d054b4221a5407c868cac2f0d7cc131" } new_table_name: "default.Table_0ec61175463840a0a8f9a373e8f3d6fa" modify_external_catalogs: true
I20250625 07:52:16.530735 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a1adb2364ae94c0aa62bc2d84fc9aded"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.531292 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1adb2364ae94c0aa62bc2d84fc9aded in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.533723 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e8d8fca59ae7412bb768e66c8ab6e77f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.534257 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d8fca59ae7412bb768e66c8ab6e77f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.535420 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_595ab735818843488af81e560f201d30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.535866 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_595ab735818843488af81e560f201d30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.536932 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ea09ec2fd2a34832ab3e312528a95048" } modify_external_catalogs: true
I20250625 07:52:16.537122 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ea09ec2fd2a34832ab3e312528a95048" } modify_external_catalogs: true
I20250625 07:52:16.538426 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 99fd0fdac6c24c0aad6f25e2f732c3d6 (table default.Table_48c1aaaa0128418cb49437cb4d940dda [id=15c681d0005742e781fe94d2c8adc9d9]) successfully deleted
W20250625 07:52:16.538648 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250175f54ba94a9fbc2455d9cf27fb2e already in progress: creating tablet
I20250625 07:52:16.539407 1160 ts_tablet_manager.cc:1918] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.540023 1160 log.cc:1199] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9e800d4982fa45f4b931985c8b52719d
I20250625 07:52:16.540019 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b9a14151fc472e956126620633be32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.540448 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_16d107cb53e442eaa68e834ce2509d3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.540783 1027 tablet_service.cc:1468] Processing CreateTablet for tablet cf175e17865e40ceb26aa955b91c2c13 (DEFAULT_TABLE table=default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.541014 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16d107cb53e442eaa68e834ce2509d3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.541673 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 32bd7cdef96946e88683e457c2f26c9b with delete_type TABLET_DATA_DELETED (replaced by tablet 2db37ece7a964f47911fe4fc7434cd07) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.542477 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 5199775108e44a9780009e4788a861ad (DEFAULT_TABLE table=default.Table_5a58b057808640f58c4b2244c66f56bf [id=9225c1ea9b174d47a1bb9421e6d251c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.542754 1315 raft_consensus.cc:513] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.543641 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfcef152c9844a59989c532bd60c07d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:16.544425 985 leader_election.cc:343] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1222f2c1027b42c6ad9627359a93d678
W20250625 07:52:16.549108 986 leader_election.cc:343] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:16.550930 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b7139335c7124ab79bc2d0a251984199 (table default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0 [id=84244d54b35c47829f8f8a92c3d47ee3]) successfully deleted
I20250625 07:52:16.552135 1166 ts_tablet_manager.cc:1918] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:16.552336 1166 log.cc:1199] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2d5b724ca63c436ba1ee776b8df3cec9
I20250625 07:52:16.552860 1166 ts_tablet_manager.cc:1939] T 2d5b724ca63c436ba1ee776b8df3cec9 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:16.553148 1160 ts_tablet_manager.cc:1939] T 9e800d4982fa45f4b931985c8b52719d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.554179 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9e800d4982fa45f4b931985c8b52719d (table default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]) successfully deleted
I20250625 07:52:16.552655 1163 tablet_bootstrap.cc:654] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.555446 1160 tablet_replica.cc:331] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.555702 1160 raft_consensus.cc:2241] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.555917 1160 raft_consensus.cc:2270] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.557022 1160 ts_tablet_manager.cc:1905] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.559152 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf175e17865e40ceb26aa955b91c2c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.562716 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b71aadf074a4a39943b2ff930909f93 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.563570 985 leader_election.cc:304] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:16.567225 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5199775108e44a9780009e4788a861ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.568723 1163 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.568969 1163 ts_tablet_manager.cc:1397] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.210s user 0.007s sys 0.003s
I20250625 07:52:16.569676 1315 leader_election.cc:290] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:16.569730 1163 raft_consensus.cc:357] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.570056 1163 raft_consensus.cc:383] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.570176 1163 raft_consensus.cc:738] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.570458 1163 consensus_queue.cc:260] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.572007 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 32bd7cdef96946e88683e457c2f26c9b with delete_type TABLET_DATA_DELETED (replaced by tablet 2db37ece7a964f47911fe4fc7434cd07) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.572225 743 tablet_service.cc:1468] Processing CreateTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.572662 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.573043 889 tablet_service.cc:1468] Processing CreateTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.573402 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.581247 1163 ts_tablet_manager.cc:1428] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20250625 07:52:16.581789 1163 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.582500 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2d5b724ca63c436ba1ee776b8df3cec9 (table default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]) successfully deleted
W20250625 07:52:16.582984 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 32bd7cdef96946e88683e457c2f26c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32bd7cdef96946e88683e457c2f26c9b already in progress: creating tablet
I20250625 07:52:16.584013 889 tablet_service.cc:1468] Processing CreateTablet for tablet fdad7dc1b27e4d87a747642a6d47230c (DEFAULT_TABLE table=default.Table_75ee0958612248248c0bfe106c759571 [id=6da9e5a92547450b83830b27727c69ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.584228 743 tablet_service.cc:1468] Processing CreateTablet for tablet fdad7dc1b27e4d87a747642a6d47230c (DEFAULT_TABLE table=default.Table_75ee0958612248248c0bfe106c759571 [id=6da9e5a92547450b83830b27727c69ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.584458 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdad7dc1b27e4d87a747642a6d47230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.584674 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdad7dc1b27e4d87a747642a6d47230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.587630 1163 tablet_bootstrap.cc:654] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.589546 1315 raft_consensus.cc:2747] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.589931 1160 ts_tablet_manager.cc:1918] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.590466 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6b71aadf074a4a39943b2ff930909f93
I20250625 07:52:16.590957 592 catalog_manager.cc:6652] ReplaceTablet: tablet 6b71aadf074a4a39943b2ff930909f93 deleted and replaced by tablet 9cc1842f257d409ba0a56873f582f023
I20250625 07:52:16.591133 1160 log.cc:1199] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:16.591567 1160 ts_tablet_manager.cc:1939] T 8e8d65c2ed5240fc923a7c7b59488e6a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.593271 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:16.594589 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:16.595232 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c263b671be384cd097041c60306034de
I20250625 07:52:16.593338 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b71aadf074a4a39943b2ff930909f93 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc1842f257d409ba0a56873f582f023) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.594497 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.594949 885 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.595592 895 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.596668 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.597081 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8e8d65c2ed5240fc923a7c7b59488e6a (table default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]) successfully deleted
I20250625 07:52:16.598758 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2f63fdf2c8464c8fb58f412263d4e5c5
W20250625 07:52:16.597822 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e289ea5553ac4ff983871ece4fc72ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e289ea5553ac4ff983871ece4fc72ee5 already in progress: creating tablet
W20250625 07:52:16.599458 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28e490ba5d08498f921b50f057b88369 consensus state has no leader
I20250625 07:52:16.599561 1160 tablet_replica.cc:331] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.600138 1160 raft_consensus.cc:2241] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:16.599370 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet cbdbd9230210455ba615d51f1a98267b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdbd9230210455ba615d51f1a98267b already in progress: creating tablet
I20250625 07:52:16.601366 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.601662 1160 raft_consensus.cc:2270] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.602946 1160 ts_tablet_manager.cc:1905] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:16.603502 851 consensus_peers.cc:487] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 -> Peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Couldn't send request to peer 191b611b772a4fda9d1cba98776138a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d5b724ca63c436ba1ee776b8df3cec9. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:16.605420 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b (DEFAULT_TABLE table=default.Table_e0ba3665554d4b36a261ce71b5fd743c [id=357521ce57d54dafae9cfd2efbe731c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.605886 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aeb7077591f466eb77e3096f6b1d74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.608505 1163 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.608664 1163 ts_tablet_manager.cc:1397] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20250625 07:52:16.608867 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.609102 757 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.609333 750 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.609519 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.609758 1163 raft_consensus.cc:357] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.609964 1163 raft_consensus.cc:383] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.610038 1163 raft_consensus.cc:738] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.610318 1163 consensus_queue.cc:260] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.613067 1163 ts_tablet_manager.cc:1428] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:16.614025 1163 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.618039 889 tablet_service.cc:1468] Processing CreateTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b (DEFAULT_TABLE table=default.Table_e0ba3665554d4b36a261ce71b5fd743c [id=357521ce57d54dafae9cfd2efbe731c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.618484 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aeb7077591f466eb77e3096f6b1d74b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:16.615374 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fc6c3be49be14ecbb3c26ef107ede015 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc6c3be49be14ecbb3c26ef107ede015 already in progress: creating tablet
W20250625 07:52:16.618749 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet cd728ffc2a274deea3fad533d1eaa6da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd728ffc2a274deea3fad533d1eaa6da already in progress: creating tablet
W20250625 07:52:16.619004 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c263b671be384cd097041c60306034de because the tablet was not found. No further retry: Not found: Tablet not found: c263b671be384cd097041c60306034de
W20250625 07:52:16.619226 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f63fdf2c8464c8fb58f412263d4e5c5 already in progress: creating tablet
I20250625 07:52:16.620172 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (DEFAULT_TABLE table=default.Table_85d42dece3ab4be5a22696c28417f194 [id=4181f3a804aa47f99e1c30e8d1307d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.620767 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2082e5e28c5c4d1fac7e40acf77f75e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.620873 895 tablet_service.cc:1468] Processing CreateTablet for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (DEFAULT_TABLE table=default.Table_85d42dece3ab4be5a22696c28417f194 [id=4181f3a804aa47f99e1c30e8d1307d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.621218 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2082e5e28c5c4d1fac7e40acf77f75e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.624008 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9babc704c99d4319bf7f3906ef371e72" } new_table_name: "default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7" modify_external_catalogs: true
I20250625 07:52:16.625043 885 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7daf759e324c1db853fdbaab76c721 (DEFAULT_TABLE table=default.Table_e0bee6f682a14fa59118497e9a616ebf [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.625401 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7daf759e324c1db853fdbaab76c721. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.627919 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b71aadf074a4a39943b2ff930909f93 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc1842f257d409ba0a56873f582f023) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.630517 747 tablet_service.cc:1468] Processing CreateTablet for tablet c263b671be384cd097041c60306034de (DEFAULT_TABLE table=default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.630990 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c263b671be384cd097041c60306034de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.632573 890 tablet_service.cc:1468] Processing CreateTablet for tablet c263b671be384cd097041c60306034de (DEFAULT_TABLE table=default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.633013 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c263b671be384cd097041c60306034de. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:16.634325 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3eea1c257b584a029efa08922547b0cc consensus state has no leader
W20250625 07:52:16.635530 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6b71aadf074a4a39943b2ff930909f93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b71aadf074a4a39943b2ff930909f93 already in progress: creating tablet
I20250625 07:52:16.637208 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7daf759e324c1db853fdbaab76c721 (DEFAULT_TABLE table=default.Table_e0bee6f682a14fa59118497e9a616ebf [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.637598 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7daf759e324c1db853fdbaab76c721. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.639704 756 tablet_service.cc:1468] Processing CreateTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb (DEFAULT_TABLE table=default.Table_4124f127e3e54adab2a8e666a1f18e48 [id=68ab97279148448991b34aeef51de39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.639806 881 tablet_service.cc:1468] Processing CreateTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb (DEFAULT_TABLE table=default.Table_4124f127e3e54adab2a8e666a1f18e48 [id=68ab97279148448991b34aeef51de39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.640161 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a07af928ff4d3d894b074ef0cf44eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.640305 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a07af928ff4d3d894b074ef0cf44eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.641264 1161 ts_tablet_manager.cc:1905] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.645025 750 tablet_service.cc:1468] Processing CreateTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.645001 884 tablet_service.cc:1468] Processing CreateTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.645383 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0657163b9a0d4f7f99f57f7fd1f32462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.645413 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0657163b9a0d4f7f99f57f7fd1f32462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.645938 1308 raft_consensus.cc:491] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.646142 1308 raft_consensus.cc:513] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.646982 1308 leader_election.cc:290] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:16.647750 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a53d8f50224460b932f4f44659336da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:16.648409 851 leader_election.cc:343] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.649049 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a53d8f50224460b932f4f44659336da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:16.649724 850 leader_election.cc:343] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.649859 850 leader_election.cc:304] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.650130 1308 raft_consensus.cc:2747] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.656625 1163 tablet_bootstrap.cc:654] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.660295 1162 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:16.660511 1162 ts_tablet_manager.cc:1397] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.207s user 0.015s sys 0.003s
I20250625 07:52:16.661099 1268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:16.661340 1162 raft_consensus.cc:357] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.661556 1162 raft_consensus.cc:383] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.661638 1162 raft_consensus.cc:738] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.661914 1162 consensus_queue.cc:260] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.662665 1162 ts_tablet_manager.cc:1428] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:16.663122 1162 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:16.664960 1160 ts_tablet_manager.cc:1918] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.665521 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:16.665951 1166 tablet_replica.cc:331] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:16.665849 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea8eb0c6e92048c482cd9244519292c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.666143 1166 raft_consensus.cc:2241] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.666323 1166 raft_consensus.cc:2270] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:16.666455 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8eb0c6e92048c482cd9244519292c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.667326 1268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:16.667642 1268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:16.668437 1160 log.cc:1199] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:16.669097 1160 ts_tablet_manager.cc:1939] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.669759 1166 ts_tablet_manager.cc:1905] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.670810 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (table default.Table_3c1cbfecab414abaa74c66f1773f817e [id=5174ca174ab24d7695eb1fddc9414c74]) successfully deleted
I20250625 07:52:16.672149 1163 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.672338 1163 ts_tablet_manager.cc:1397] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20250625 07:52:16.673295 1163 raft_consensus.cc:357] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.673480 1158 tablet_bootstrap.cc:492] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:16.673538 1163 raft_consensus.cc:383] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.673650 1158 ts_tablet_manager.cc:1397] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.357s user 0.018s sys 0.015s
I20250625 07:52:16.673764 1163 raft_consensus.cc:738] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.674044 1163 consensus_queue.cc:260] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.676492 1166 ts_tablet_manager.cc:1918] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.676632 1166 log.cc:1199] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3338d1c708b4421c8501740da89afec5
I20250625 07:52:16.677038 1166 ts_tablet_manager.cc:1939] T 3338d1c708b4421c8501740da89afec5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:16.677027 1163 ts_tablet_manager.cc:1428] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:16.677408 1163 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.677654 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c928e3f5ce01487c976a35b088cbe9ad" } new_table_name: "default.Table_5a2522dd8ce349ec8bfa57ba2d10b114" modify_external_catalogs: true
I20250625 07:52:16.678967 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aacfa5f603c8417288db11b6ae09f880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.679476 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aacfa5f603c8417288db11b6ae09f880 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:16.680030 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6239713a586d43c28c101ba2447d579d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6239713a586d43c28c101ba2447d579d already in progress: creating tablet
I20250625 07:52:16.680603 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_41e55514978543459d3cbec553136e88" } new_table_name: "default.Table_c96df891fb1343df8cc90ee2304287fb" modify_external_catalogs: true
W20250625 07:52:16.680747 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6e5a207d515449e298936d8bb532437a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e5a207d515449e298936d8bb532437a already in progress: creating tablet
I20250625 07:52:16.681039 1160 tablet_replica.cc:331] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.681239 1160 raft_consensus.cc:2241] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.681458 1160 raft_consensus.cc:2270] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.681800 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1437af2855fc49d6b981d94261019399"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:16.682077 1163 tablet_bootstrap.cc:654] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.682202 1160 ts_tablet_manager.cc:1905] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:16.682294 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1437af2855fc49d6b981d94261019399 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.683354 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c24058d938ab46a3a19827cf537af456" } modify_external_catalogs: true
I20250625 07:52:16.683516 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c24058d938ab46a3a19827cf537af456" } modify_external_catalogs: true
I20250625 07:52:16.684026 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b2a733ce9ffb4a74b2c8f1716dc57187" } new_table_name: "default.Table_093618aa2b3243bd99e0f25c8e738c26" modify_external_catalogs: true
I20250625 07:52:16.690878 1158 raft_consensus.cc:357] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.691084 1158 raft_consensus.cc:383] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.691239 1158 raft_consensus.cc:738] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.691454 1158 consensus_queue.cc:260] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.692320 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 49697a10cdac4c8ebb69f10f4c3437ec from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.695125 1160 ts_tablet_manager.cc:1918] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.698217 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3338d1c708b4421c8501740da89afec5 (table default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]) successfully deleted
I20250625 07:52:16.700145 1158 ts_tablet_manager.cc:1428] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250625 07:52:16.696447 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:16.705979 1160 log.cc:1199] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b769ac57346a427abd22214eebc0d324
I20250625 07:52:16.706506 1160 ts_tablet_manager.cc:1939] T b769ac57346a427abd22214eebc0d324 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.707818 1160 tablet_replica.cc:331] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.707943 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b769ac57346a427abd22214eebc0d324 (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:16.708065 1160 raft_consensus.cc:2241] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.708259 1160 raft_consensus.cc:2270] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.709023 1160 ts_tablet_manager.cc:1905] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.714643 1162 tablet_bootstrap.cc:654] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.718910 1158 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:16.720788 1158 tablet_bootstrap.cc:654] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:16.721843 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 181143us
I20250625 07:52:16.722720 1160 ts_tablet_manager.cc:1918] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.722887 1160 log.cc:1199] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:16.723378 1160 ts_tablet_manager.cc:1939] T 6b1eaf0266c04873b185c1c0c2da201c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.724525 1160 tablet_replica.cc:331] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.725116 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6b1eaf0266c04873b185c1c0c2da201c (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
I20250625 07:52:16.726337 1027 tablet_service.cc:1468] Processing CreateTablet for tablet fdad7dc1b27e4d87a747642a6d47230c (DEFAULT_TABLE table=default.Table_75ee0958612248248c0bfe106c759571 [id=6da9e5a92547450b83830b27727c69ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.726763 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdad7dc1b27e4d87a747642a6d47230c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.728268 1160 raft_consensus.cc:2241] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.729259 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b (DEFAULT_TABLE table=default.Table_e0ba3665554d4b36a261ce71b5fd743c [id=357521ce57d54dafae9cfd2efbe731c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.729616 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aeb7077591f466eb77e3096f6b1d74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.730633 1160 raft_consensus.cc:2270] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.732008 1160 ts_tablet_manager.cc:1905] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.733148 1014 tablet_service.cc:1468] Processing CreateTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.737180 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.744223 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f16f5e9c3f6d4110baa383f8551d2f91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.744876 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f16f5e9c3f6d4110baa383f8551d2f91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.746482 1160 ts_tablet_manager.cc:1918] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:16.746959 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fd497aa0b8c4838906524f4423c3756 consensus state has no leader
I20250625 07:52:16.747869 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b71aadf074a4a39943b2ff930909f93 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc1842f257d409ba0a56873f582f023) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.748390 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.748787 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.752102 1160 log.cc:1199] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3c8c317c10e14332a3b3919808e4a512
I20250625 07:52:16.752972 1160 ts_tablet_manager.cc:1939] T 3c8c317c10e14332a3b3919808e4a512 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.753002 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:16.753175 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 019959e9abed46fbbbb4755cb01cef44 consensus state has no leader
I20250625 07:52:16.753602 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_406f7f4727764ec1be3f67c739f08a6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.754257 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_406f7f4727764ec1be3f67c739f08a6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.754418 1161 ts_tablet_manager.cc:1918] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.754549 1161 log.cc:1199] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c44b0ef2ac60457bb56abead72f0182a
I20250625 07:52:16.754936 1161 ts_tablet_manager.cc:1939] T c44b0ef2ac60457bb56abead72f0182a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:16.754968 1160 tablet_replica.cc:331] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.755178 1160 raft_consensus.cc:2241] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.755347 1160 raft_consensus.cc:2270] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.755374 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3c8c317c10e14332a3b3919808e4a512 (table default.Table_ce2222183ca64261b57847f5be67d81a [id=152fbc75a19b4eacb4909704af909443]) successfully deleted
I20250625 07:52:16.755656 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.756040 1160 ts_tablet_manager.cc:1905] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.758641 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_82cd9b0cdb87402c90df8c979e56570a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.759558 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82cd9b0cdb87402c90df8c979e56570a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.760803 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 572b063d66434e4796f6b86f8221e3c9 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.761266 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (DEFAULT_TABLE table=default.Table_85d42dece3ab4be5a22696c28417f194 [id=4181f3a804aa47f99e1c30e8d1307d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.762621 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 445bfed532004108a12c8fd78370c1cb from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.765447 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c263b671be384cd097041c60306034de (DEFAULT_TABLE table=default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.768550 985 leader_election.cc:343] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:16.768847 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2082e5e28c5c4d1fac7e40acf77f75e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.769800 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb (DEFAULT_TABLE table=default.Table_4124f127e3e54adab2a8e666a1f18e48 [id=68ab97279148448991b34aeef51de39a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:16.770795 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4729a62b7434cc0bcd1eecbfe838faf has no consensus state
I20250625 07:52:16.771430 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 (DEFAULT_TABLE table=default.Table_dac905b911e84c5cad6ab113b3907866 [id=904f631d8b704232b9182be1fa4aff8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.778688 1160 ts_tablet_manager.cc:1918] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.778873 1160 log.cc:1199] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/95a763abda074d358f176fd7e7be0e5c
I20250625 07:52:16.779300 1160 ts_tablet_manager.cc:1939] T 95a763abda074d358f176fd7e7be0e5c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.779899 1308 raft_consensus.cc:491] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:16.785698 1308 raft_consensus.cc:513] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.786391 1308 leader_election.cc:290] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:16.787125 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 95a763abda074d358f176fd7e7be0e5c (table default.Table_1e843117005043efadef968ca11c482f [id=44ba0323288e45269c5dda7cd7f60f84]) successfully deleted
I20250625 07:52:16.786417 1160 tablet_replica.cc:331] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.787348 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0452e73865964f709310fd22ac39a5f0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:16.787560 1160 raft_consensus.cc:2241] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.787703 1160 raft_consensus.cc:2270] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.787787 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0452e73865964f709310fd22ac39a5f0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:16.788755 1160 ts_tablet_manager.cc:1905] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:16.788872 851 leader_election.cc:343] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:16.789674 850 leader_election.cc:343] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:16.789788 850 leader_election.cc:304] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:16.790055 1308 raft_consensus.cc:2747] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.794874 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7daf759e324c1db853fdbaab76c721 (DEFAULT_TABLE table=default.Table_e0bee6f682a14fa59118497e9a616ebf [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.801599 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 206) took 2274 ms. Trace:
I20250625 07:52:16.801919 1161 rpcz_store.cc:276] 0625 07:52:14.527059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.527650 (+ 591us) service_pool.cc:225] Handling call
0625 07:52:16.448568 (+1920918us) ts_tablet_manager.cc:1145] Deleting tablet c44b0ef2ac60457bb56abead72f0182a
0625 07:52:16.448576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.689435 (+240859us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.754272 (+ 64837us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.801500 (+ 47228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920695}
I20250625 07:52:16.802803 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 207) took 2275 ms. Trace:
I20250625 07:52:16.803082 1161 rpcz_store.cc:276] 0625 07:52:14.527767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.535283 (+ 7516us) service_pool.cc:225] Handling call
0625 07:52:16.802586 (+2267303us) ts_tablet_manager.cc:1145] Deleting tablet 250175f54ba94a9fbc2455d9cf27fb2e
0625 07:52:16.802595 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.802773 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267068}
I20250625 07:52:16.803776 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 209) took 2257 ms. Trace:
I20250625 07:52:16.804101 1160 ts_tablet_manager.cc:1918] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.804015 1161 rpcz_store.cc:276] 0625 07:52:14.545905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.549039 (+ 3134us) service_pool.cc:225] Handling call
0625 07:52:16.803575 (+2254536us) ts_tablet_manager.cc:1145] Deleting tablet 5c28a064006145eea000e74da10d0794
0625 07:52:16.803584 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.803719 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254262}
I20250625 07:52:16.804536 1161 tablet_replica.cc:331] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:16.804924 1161 raft_consensus.cc:2241] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.805229 1161 raft_consensus.cc:2270] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.806236 1160 log.cc:1199] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7a53d8f50224460b932f4f44659336da
I20250625 07:52:16.806354 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c263b671be384cd097041c60306034de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.807044 1160 ts_tablet_manager.cc:1939] T 7a53d8f50224460b932f4f44659336da P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.808240 1160 tablet_replica.cc:331] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.808421 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7a53d8f50224460b932f4f44659336da (table default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]) successfully deleted
I20250625 07:52:16.808512 1160 raft_consensus.cc:2241] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:16.809232 1160 raft_consensus.cc:2270] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.809290 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e75429a686be4882a9676251755c548b" } modify_external_catalogs: true
I20250625 07:52:16.809468 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e75429a686be4882a9676251755c548b" } modify_external_catalogs: true
I20250625 07:52:16.810200 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_28a8c56dcb9b4bf093c733607ff00d69" } modify_external_catalogs: true
I20250625 07:52:16.810385 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_28a8c56dcb9b4bf093c733607ff00d69" } modify_external_catalogs: true
I20250625 07:52:16.810905 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b047e3c49b234a66991bed76ac01e6c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.811527 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b047e3c49b234a66991bed76ac01e6c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.812134 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e34ba3eabd8f4bafba2121af2e4ef24a" } new_table_name: "default.Table_2384bcb9ddc1469daf7ed5f61f44a71e" modify_external_catalogs: true
I20250625 07:52:16.812579 1160 ts_tablet_manager.cc:1905] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.813158 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a07af928ff4d3d894b074ef0cf44eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.814050 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_525806c470a840e8b54ce05f09aaf215"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.814539 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_525806c470a840e8b54ce05f09aaf215 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.815790 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dfe7357f10e546fa8c2a6bcc522beb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.816206 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfe7357f10e546fa8c2a6bcc522beb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.817260 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0657163b9a0d4f7f99f57f7fd1f32462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.817746 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a881661cd92a4541a73b939c4532acc7" } modify_external_catalogs: true
I20250625 07:52:16.817931 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a881661cd92a4541a73b939c4532acc7" } modify_external_catalogs: true
I20250625 07:52:16.818512 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4a0e01d2875e4b639d653e035aa60679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.818997 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a0e01d2875e4b639d653e035aa60679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:16.820384 1160 ts_tablet_manager.cc:1918] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:16.825261 985 leader_election.cc:304] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:16.825645 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f84acd45845c4917b7dc8c5ad5232897 consensus state has no leader
I20250625 07:52:16.825632 1373 raft_consensus.cc:2747] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:16.826539 1158 tablet_bootstrap.cc:492] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:16.826701 1158 ts_tablet_manager.cc:1397] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.000s
I20250625 07:52:16.827457 1158 raft_consensus.cc:357] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.827638 1158 raft_consensus.cc:383] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.827724 1158 raft_consensus.cc:738] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.828003 1158 consensus_queue.cc:260] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:16.828622 1158 ts_tablet_manager.cc:1428] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20250625 07:52:16.828886 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 60375.5us
I20250625 07:52:16.829177 1160 log.cc:1199] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2d5b724ca63c436ba1ee776b8df3cec9
I20250625 07:52:16.829953 1160 ts_tablet_manager.cc:1939] T 2d5b724ca63c436ba1ee776b8df3cec9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.830791 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c44b0ef2ac60457bb56abead72f0182a (table default.Table_808e1666e3d347168df6c5a4301fc6d8 [id=f87841f2973841248d450e263c7a5d83]) successfully deleted
W20250625 07:52:16.831256 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 250175f54ba94a9fbc2455d9cf27fb2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250175f54ba94a9fbc2455d9cf27fb2e already in progress: creating tablet
W20250625 07:52:16.831703 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5c28a064006145eea000e74da10d0794 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c28a064006145eea000e74da10d0794 already in progress: creating tablet
I20250625 07:52:16.831954 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2d5b724ca63c436ba1ee776b8df3cec9 (table default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]) successfully deleted
I20250625 07:52:16.831703 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4f95753979a34ea994409ba5978b7186" } modify_external_catalogs: true
I20250625 07:52:16.832342 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4f95753979a34ea994409ba5978b7186" } modify_external_catalogs: true
I20250625 07:52:16.833024 1160 tablet_replica.cc:331] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.833680 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7daf759e324c1db853fdbaab76c721. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.833659 1160 raft_consensus.cc:2241] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.833880 1160 raft_consensus.cc:2270] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.835142 1160 ts_tablet_manager.cc:1905] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.841812 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_71a35f9021c7495ca3d322bf4adf02d6" } new_table_name: "default.Table_4bb58e159ce944079a5760e1acb82e57" modify_external_catalogs: true
I20250625 07:52:16.849462 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet d51a898202154881ab4ad41da1d519a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.850693 1160 ts_tablet_manager.cc:1918] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.854811 1162 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:16.855046 1162 ts_tablet_manager.cc:1397] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.192s user 0.004s sys 0.003s
I20250625 07:52:16.855665 1160 log.cc:1199] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/32bd7cdef96946e88683e457c2f26c9b
I20250625 07:52:16.855955 1162 raft_consensus.cc:357] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.856225 1162 raft_consensus.cc:383] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.856367 1162 raft_consensus.cc:738] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.856374 1160 ts_tablet_manager.cc:1939] T 32bd7cdef96946e88683e457c2f26c9b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.856719 1162 consensus_queue.cc:260] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.858508 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 32bd7cdef96946e88683e457c2f26c9b (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]) successfully deleted
I20250625 07:52:16.859182 1160 tablet_replica.cc:331] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.859604 1162 ts_tablet_manager.cc:1428] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250625 07:52:16.859804 1160 raft_consensus.cc:2241] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.859982 1162 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:16.860369 1160 raft_consensus.cc:2270] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.861385 1160 ts_tablet_manager.cc:1905] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.865547 1158 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:16.868754 1158 tablet_bootstrap.cc:654] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.876574 1161 ts_tablet_manager.cc:1905] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.878891 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 239e99c6b91d41d68a213a4c8d79470f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.880967 1160 ts_tablet_manager.cc:1918] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.881758 1160 log.cc:1199] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6b71aadf074a4a39943b2ff930909f93
I20250625 07:52:16.882416 1160 ts_tablet_manager.cc:1939] T 6b71aadf074a4a39943b2ff930909f93 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.884331 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6b71aadf074a4a39943b2ff930909f93 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:16.885849 1160 tablet_replica.cc:331] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.886451 1160 raft_consensus.cc:2241] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.887027 1160 raft_consensus.cc:2270] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.888379 1160 ts_tablet_manager.cc:1905] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.888996 1162 tablet_bootstrap.cc:654] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.890087 1158 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:16.890254 1158 ts_tablet_manager.cc:1397] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20250625 07:52:16.891053 1158 raft_consensus.cc:357] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.891237 1158 raft_consensus.cc:383] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.891320 1158 raft_consensus.cc:738] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.891533 1158 consensus_queue.cc:260] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:16.902168 1160 ts_tablet_manager.cc:1918] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.903453 1161 ts_tablet_manager.cc:1918] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.903628 1161 log.cc:1199] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/95fce53f9a654fcaa8a52e5733c1b74f
I20250625 07:52:16.904023 1161 ts_tablet_manager.cc:1939] T 95fce53f9a654fcaa8a52e5733c1b74f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:16.904371 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 216) took 1950 ms. Trace:
I20250625 07:52:16.904459 1161 rpcz_store.cc:276] 0625 07:52:14.954216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.954299 (+ 83us) service_pool.cc:225] Handling call
0625 07:52:16.804429 (+1850130us) ts_tablet_manager.cc:1145] Deleting tablet 95fce53f9a654fcaa8a52e5733c1b74f
0625 07:52:16.804463 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:16.899669 (+ 95206us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.903353 (+ 3684us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:16.904323 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849842}
I20250625 07:52:16.908447 1160 log.cc:1199] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0452e73865964f709310fd22ac39a5f0
I20250625 07:52:16.908478 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7db12504a1a54add920561dae27d2d2b (DEFAULT_TABLE table=default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.908965 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db12504a1a54add920561dae27d2d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.909317 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7db12504a1a54add920561dae27d2d2b (DEFAULT_TABLE table=default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.909722 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db12504a1a54add920561dae27d2d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.910392 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 7db12504a1a54add920561dae27d2d2b (DEFAULT_TABLE table=default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.910760 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db12504a1a54add920561dae27d2d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.911304 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdc3220d53142faae41fd1f1500de56 (DEFAULT_TABLE table=default.Table_595ab735818843488af81e560f201d30 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.911661 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdc3220d53142faae41fd1f1500de56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.913024 747 tablet_service.cc:1468] Processing CreateTablet for tablet 15766b87fb1b4620970d5fab21465cc4 (DEFAULT_TABLE table=default.Table_e8d8fca59ae7412bb768e66c8ab6e77f [id=d4be884fab4f4468a41fb5a7eff91ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.913401 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15766b87fb1b4620970d5fab21465cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.913368 890 tablet_service.cc:1468] Processing CreateTablet for tablet 15766b87fb1b4620970d5fab21465cc4 (DEFAULT_TABLE table=default.Table_e8d8fca59ae7412bb768e66c8ab6e77f [id=d4be884fab4f4468a41fb5a7eff91ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.913712 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15766b87fb1b4620970d5fab21465cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.917886 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdc3220d53142faae41fd1f1500de56 (DEFAULT_TABLE table=default.Table_595ab735818843488af81e560f201d30 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.918277 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 15766b87fb1b4620970d5fab21465cc4 (DEFAULT_TABLE table=default.Table_e8d8fca59ae7412bb768e66c8ab6e77f [id=d4be884fab4f4468a41fb5a7eff91ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.918283 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdc3220d53142faae41fd1f1500de56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.923874 1160 ts_tablet_manager.cc:1939] T 0452e73865964f709310fd22ac39a5f0 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:16.924674 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15766b87fb1b4620970d5fab21465cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.925004 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8fdc3220d53142faae41fd1f1500de56 (DEFAULT_TABLE table=default.Table_595ab735818843488af81e560f201d30 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.925487 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fdc3220d53142faae41fd1f1500de56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.926331 750 tablet_service.cc:1468] Processing CreateTablet for tablet af8c114c1c464bcc95dbd287c88976d6 (DEFAULT_TABLE table=default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.926698 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c114c1c464bcc95dbd287c88976d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.927883 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:16.928233 1158 ts_tablet_manager.cc:1428] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20250625 07:52:16.929651 1158 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:16.933823 890 tablet_service.cc:1468] Processing CreateTablet for tablet af8c114c1c464bcc95dbd287c88976d6 (DEFAULT_TABLE table=default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.934257 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9cc1842f257d409ba0a56873f582f023 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.934484 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c114c1c464bcc95dbd287c88976d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.934634 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc1842f257d409ba0a56873f582f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.936975 1027 tablet_service.cc:1468] Processing CreateTablet for tablet af8c114c1c464bcc95dbd287c88976d6 (DEFAULT_TABLE table=default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.937474 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8c114c1c464bcc95dbd287c88976d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.941895 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0452e73865964f709310fd22ac39a5f0 (table default.Table_c589a7d832eb4e7fb74f00c8fcbaa680 [id=7ce5c0adc972450bb89be40d18e4cc30]) successfully deleted
W20250625 07:52:16.943346 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fc6c3be49be14ecbb3c26ef107ede015 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc6c3be49be14ecbb3c26ef107ede015 already in progress: creating tablet
I20250625 07:52:16.943293 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7fc614d790ab4feaa0e3167c5de7cffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:16.943899 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fc614d790ab4feaa0e3167c5de7cffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:16.944849 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet cd728ffc2a274deea3fad533d1eaa6da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd728ffc2a274deea3fad533d1eaa6da already in progress: creating tablet
W20250625 07:52:16.948554 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c263b671be384cd097041c60306034de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c263b671be384cd097041c60306034de already in progress: creating tablet
W20250625 07:52:16.949726 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f63fdf2c8464c8fb58f412263d4e5c5 already in progress: creating tablet
I20250625 07:52:16.951932 1160 tablet_replica.cc:331] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:16.952554 1160 raft_consensus.cc:2241] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:16.953142 1160 raft_consensus.cc:2270] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:16.953583 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 49697a10cdac4c8ebb69f10f4c3437ec from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:16.954646 1160 ts_tablet_manager.cc:1905] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:16.955988 1158 tablet_bootstrap.cc:654] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.964653 890 tablet_service.cc:1468] Processing CreateTablet for tablet 9cc1842f257d409ba0a56873f582f023 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.965334 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc1842f257d409ba0a56873f582f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.967411 743 tablet_service.cc:1468] Processing CreateTablet for tablet 716e0c4dab584788bbb7631b4ca934bf (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.967828 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716e0c4dab584788bbb7631b4ca934bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.968853 1163 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:16.969159 1163 ts_tablet_manager.cc:1397] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.292s user 0.069s sys 0.087s
I20250625 07:52:16.969269 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 9cc1842f257d409ba0a56873f582f023 (DEFAULT_TABLE table=default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.969691 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc1842f257d409ba0a56873f582f023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.970430 1163 raft_consensus.cc:357] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.970981 1163 raft_consensus.cc:383] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:16.971122 1163 raft_consensus.cc:738] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:16.971503 1163 consensus_queue.cc:260] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:16.971580 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_91c2085052354e22a19051706e2493a5" } new_table_name: "default.Table_5d201449036c4c1a8351c46d645de96c" modify_external_catalogs: true
I20250625 07:52:16.974689 1163 ts_tablet_manager.cc:1428] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:16.975224 1163 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:16.978188 889 tablet_service.cc:1468] Processing CreateTablet for tablet 716e0c4dab584788bbb7631b4ca934bf (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.978809 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716e0c4dab584788bbb7631b4ca934bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.979861 747 tablet_service.cc:1468] Processing CreateTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 (DEFAULT_TABLE table=default.Table_66abf1d5bf9b498aacef39657e91e79c [id=45c3c59676b0417fb034a9ffd06fee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.980247 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b87309b76f4f0e9dde120d9f755c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.986651 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 716e0c4dab584788bbb7631b4ca934bf (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.987777 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 716e0c4dab584788bbb7631b4ca934bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.988175 1163 tablet_bootstrap.cc:654] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:16.989465 747 tablet_service.cc:1468] Processing CreateTablet for tablet 2db37ece7a964f47911fe4fc7434cd07 (DEFAULT_TABLE table=default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.990016 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db37ece7a964f47911fe4fc7434cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.993317 1160 ts_tablet_manager.cc:1918] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:16.998137 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 (DEFAULT_TABLE table=default.Table_66abf1d5bf9b498aacef39657e91e79c [id=45c3c59676b0417fb034a9ffd06fee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.997738 895 tablet_service.cc:1468] Processing CreateTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 (DEFAULT_TABLE table=default.Table_66abf1d5bf9b498aacef39657e91e79c [id=45c3c59676b0417fb034a9ffd06fee91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:16.998823 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b87309b76f4f0e9dde120d9f755c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:16.999107 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b87309b76f4f0e9dde120d9f755c44. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:17.001644 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet faf049714a7e4d7eb947c29f272bbb7c consensus state has no leader
I20250625 07:52:17.003628 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_75ee0958612248248c0bfe106c759571" } new_table_name: "default.Table_a3f6090fd2ea47c1a84f1e6b83a22af8" modify_external_catalogs: true
I20250625 07:52:17.004917 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:17.005709 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bebb528a00504a7e80d68cec36278f45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.006261 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bebb528a00504a7e80d68cec36278f45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.007613 1161 tablet_replica.cc:331] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.007860 1161 raft_consensus.cc:2241] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.008895 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.009492 1160 log.cc:1199] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c6031b9695cd4a99bd0e5391663170ff
W20250625 07:52:17.009598 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 06e4bf78218f4b82b4f16f59b1704256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e4bf78218f4b82b4f16f59b1704256 already in progress: creating tablet
I20250625 07:52:17.010169 1160 ts_tablet_manager.cc:1939] T c6031b9695cd4a99bd0e5391663170ff P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.012876 1163 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.013136 1163 ts_tablet_manager.cc:1397] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20250625 07:52:17.014211 1163 raft_consensus.cc:357] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.014534 1163 raft_consensus.cc:383] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.014694 1163 raft_consensus.cc:738] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.015237 1163 consensus_queue.cc:260] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.017289 1163 ts_tablet_manager.cc:1428] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:17.017786 1163 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.022104 1163 tablet_bootstrap.cc:654] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.024807 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 29f2997d1d714100950383e6af04f315
I20250625 07:52:17.024930 1161 raft_consensus.cc:2270] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.025550 1161 ts_tablet_manager.cc:1905] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.027021 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.029552 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 29f2997d1d714100950383e6af04f315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f2997d1d714100950383e6af04f315 already in progress: creating tablet
I20250625 07:52:17.031734 756 tablet_service.cc:1468] Processing CreateTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf (DEFAULT_TABLE table=default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.031865 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_baabb7e0fd1e496bbe725e5e00574146"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.032332 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4729a62b7434cc0bcd1eecbfe838faf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:17.032506 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baabb7e0fd1e496bbe725e5e00574146 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.041956 1163 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.042230 1163 ts_tablet_manager.cc:1397] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.012s
I20250625 07:52:17.043081 1163 raft_consensus.cc:357] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.043401 1163 raft_consensus.cc:383] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.043485 1158 tablet_bootstrap.cc:492] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.043519 1163 raft_consensus.cc:738] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.043671 1158 ts_tablet_manager.cc:1397] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.003s
I20250625 07:52:17.043920 1163 consensus_queue.cc:260] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.045811 1163 ts_tablet_manager.cc:1428] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250625 07:52:17.044194 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 46716.8us
I20250625 07:52:17.047084 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.048657 895 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.049638 1160 tablet_replica.cc:331] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.049917 1160 raft_consensus.cc:2241] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.050146 1160 raft_consensus.cc:2270] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.051178 1160 ts_tablet_manager.cc:1905] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.052534 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.053332 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2db37ece7a964f47911fe4fc7434cd07 (DEFAULT_TABLE table=default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.053489 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 49697a10cdac4c8ebb69f10f4c3437ec
I20250625 07:52:17.053974 595 catalog_manager.cc:6652] ReplaceTablet: tablet 49697a10cdac4c8ebb69f10f4c3437ec deleted and replaced by tablet a41ffd566593499299aba5513ccb2f30
I20250625 07:52:17.054214 595 master_service.cc:948] ReplaceTablet: received request to replace tablet a0518fbc7f52454296ae79a3c92ef800 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.054725 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.054715 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5aad7032c870473ab9869bb56e68b932" } new_table_name: "default.Table_2f89a55bc2694564b94bb87f42761d2a" modify_external_catalogs: true
I20250625 07:52:17.054836 1025 tablet_service.cc:1468] Processing CreateTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf (DEFAULT_TABLE table=default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.055210 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2db37ece7a964f47911fe4fc7434cd07 (DEFAULT_TABLE table=default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.055735 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db37ece7a964f47911fe4fc7434cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.056043 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:17.056668 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_be7ed090191c4152b6e6bf3651b6988b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.057202 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be7ed090191c4152b6e6bf3651b6988b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.056012 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 95fce53f9a654fcaa8a52e5733c1b74f (table default.Table_4347d3ef74664e59b97f0e68e9abcfc0 [id=c6a3988b7eb347bb9d72ea73242503b7]) successfully deleted
I20250625 07:52:17.058528 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c6031b9695cd4a99bd0e5391663170ff (table default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]) successfully deleted
I20250625 07:52:17.058597 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8adb7a38211942288ca93280cd05d4f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.059125 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8adb7a38211942288ca93280cd05d4f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.059191 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.059553 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.060614 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.061026 890 tablet_service.cc:1468] Processing CreateTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf (DEFAULT_TABLE table=default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:17.061208 583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477): Couldn't send request to peer 00a3faeffaa4422ea73b98ee74768a4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:17.061509 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4729a62b7434cc0bcd1eecbfe838faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.061566 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.062229 1162 tablet_bootstrap.cc:492] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:17.062289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0bbd21d4aa41b2b92473bffe7dc05a already in progress: creating tablet
I20250625 07:52:17.062384 1162 ts_tablet_manager.cc:1397] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.202s user 0.001s sys 0.003s
I20250625 07:52:17.063087 1162 raft_consensus.cc:357] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.063263 1162 raft_consensus.cc:383] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.063324 1162 raft_consensus.cc:738] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.063584 1162 consensus_queue.cc:260] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.064946 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.065214 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.067373 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.068006 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 49697a10cdac4c8ebb69f10f4c3437ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49697a10cdac4c8ebb69f10f4c3437ec already in progress: creating tablet
I20250625 07:52:17.068634 1162 ts_tablet_manager.cc:1428] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:17.071095 1162 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.074116 1162 tablet_bootstrap.cc:654] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.075263 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:17.076115 1158 raft_consensus.cc:357] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.076391 1158 raft_consensus.cc:383] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.076483 1158 raft_consensus.cc:738] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.076800 1158 consensus_queue.cc:260] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.081557 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.082653 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bbe5c37fc9bd49c685de8f469a7af0c6
I20250625 07:52:17.083374 747 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.084224 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bbe5c37fc9bd49c685de8f469a7af0c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe5c37fc9bd49c685de8f469a7af0c6 already in progress: creating tablet
I20250625 07:52:17.085053 1163 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.087581 895 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.088454 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.089293 1160 ts_tablet_manager.cc:1918] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.089510 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.089628 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.089709 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.089779 1163 tablet_bootstrap.cc:654] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.090662 1160 log.cc:1199] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e76df9405d91488dad0cfa7f8755bb1c
I20250625 07:52:17.093667 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db37ece7a964f47911fe4fc7434cd07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.093995 1160 ts_tablet_manager.cc:1939] T e76df9405d91488dad0cfa7f8755bb1c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.094272 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4729a62b7434cc0bcd1eecbfe838faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.095674 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ee01c47b2df2467eadab63a9cbfb4b6a" } new_table_name: "default.Table_f6e3be4354c248d8abb373c9f0b5c6c9" modify_external_catalogs: true
I20250625 07:52:17.096899 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_75a5d24d45144619ab16307ddb8a5807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.097458 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75a5d24d45144619ab16307ddb8a5807 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.098728 1160 tablet_replica.cc:331] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.098680 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5ad0a090d681485683aa607f6f29b2b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.099144 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad0a090d681485683aa607f6f29b2b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.099192 1160 raft_consensus.cc:2241] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.099500 1160 raft_consensus.cc:2270] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.100205 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b020ff9f04b4d1796af4ff585cc04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.100528 1166 tablet_replica.cc:331] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.100538 1160 ts_tablet_manager.cc:1905] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.100757 1166 raft_consensus.cc:2241] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.100960 1166 raft_consensus.cc:2270] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.101049 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b3488a9fb19452dac3241170dd2fee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.102473 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e76df9405d91488dad0cfa7f8755bb1c (table default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]) successfully deleted
W20250625 07:52:17.102743 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 29f2997d1d714100950383e6af04f315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f2997d1d714100950383e6af04f315 already in progress: creating tablet
W20250625 07:52:17.102996 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 49697a10cdac4c8ebb69f10f4c3437ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49697a10cdac4c8ebb69f10f4c3437ec already in progress: creating tablet
W20250625 07:52:17.103210 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0bbd21d4aa41b2b92473bffe7dc05a already in progress: creating tablet
W20250625 07:52:17.103405 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 06e4bf78218f4b82b4f16f59b1704256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e4bf78218f4b82b4f16f59b1704256 already in progress: creating tablet
W20250625 07:52:17.103638 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bbe5c37fc9bd49c685de8f469a7af0c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe5c37fc9bd49c685de8f469a7af0c6 already in progress: creating tablet
I20250625 07:52:17.104856 1373 raft_consensus.cc:491] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.105021 1373 raft_consensus.cc:513] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.106187 1373 leader_election.cc:290] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:17.107362 986 leader_election.cc:343] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6b3488a9fb19452dac3241170dd2fee2
I20250625 07:52:17.108072 1163 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.108351 1163 ts_tablet_manager.cc:1397] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.003s
I20250625 07:52:17.109496 1163 raft_consensus.cc:357] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.109819 1163 raft_consensus.cc:383] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.110000 1163 raft_consensus.cc:738] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.110358 1163 consensus_queue.cc:260] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
W20250625 07:52:17.111145 985 leader_election.cc:343] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6b3488a9fb19452dac3241170dd2fee2
I20250625 07:52:17.111335 985 leader_election.cc:304] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:17.111433 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d76681f162074b67a6b886887b5a3d61 consensus state has no leader
I20250625 07:52:17.111903 1373 raft_consensus.cc:2747] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.100539 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.113372 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d375845276c6425baf08c8e94b318afc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.113546 1166 ts_tablet_manager.cc:1905] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:17.113984 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d375845276c6425baf08c8e94b318afc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.116818 1163 ts_tablet_manager.cc:1428] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250625 07:52:17.120113 1166 ts_tablet_manager.cc:1918] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.120301 1166 log.cc:1199] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2b020ff9f04b4d1796af4ff585cc04aa
I20250625 07:52:17.121388 1163 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.123809 1166 ts_tablet_manager.cc:1939] T 2b020ff9f04b4d1796af4ff585cc04aa P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.124606 1163 tablet_bootstrap.cc:654] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.124934 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2b020ff9f04b4d1796af4ff585cc04aa (table default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]) successfully deleted
W20250625 07:52:17.125718 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a3670cadfa2841949705c65c9b6c0927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3670cadfa2841949705c65c9b6c0927 already in progress: creating tablet
I20250625 07:52:17.129119 1158 ts_tablet_manager.cc:1428] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.085s user 0.003s sys 0.000s
W20250625 07:52:17.129406 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdad7dc1b27e4d87a747642a6d47230c consensus state has no leader
I20250625 07:52:17.131189 1158 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.130482 1160 ts_tablet_manager.cc:1918] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.131856 1160 log.cc:1199] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4c3c180785b64f7d85863c14cfd09ccc
I20250625 07:52:17.132645 1160 ts_tablet_manager.cc:1939] T 4c3c180785b64f7d85863c14cfd09ccc P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.134297 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4c3c180785b64f7d85863c14cfd09ccc (table default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]) successfully deleted
I20250625 07:52:17.134536 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f6885e8b33124109bb5ab1f1128522de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.135176 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6885e8b33124109bb5ab1f1128522de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.135799 1160 tablet_replica.cc:331] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.136049 1160 raft_consensus.cc:2241] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.136277 1160 raft_consensus.cc:2270] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.138089 1160 ts_tablet_manager.cc:1905] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.139609 1158 tablet_bootstrap.cc:654] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.141889 1162 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.142118 1163 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.142141 1162 ts_tablet_manager.cc:1397] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.003s
I20250625 07:52:17.142544 1163 ts_tablet_manager.cc:1397] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20250625 07:52:17.142980 1162 raft_consensus.cc:357] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.143241 1162 raft_consensus.cc:383] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.143358 1162 raft_consensus.cc:738] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.143656 1162 consensus_queue.cc:260] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.143604 1163 raft_consensus.cc:357] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.144069 1163 raft_consensus.cc:383] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.144230 1163 raft_consensus.cc:738] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.144655 1163 consensus_queue.cc:260] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.148188 1162 ts_tablet_manager.cc:1428] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 07:52:17.150031 1163 ts_tablet_manager.cc:1428] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:17.151319 1162 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.154521 1161 ts_tablet_manager.cc:1918] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.154695 1163 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.154737 1161 log.cc:1199] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/51279fb42ae8480b8511432cb4f808ab
I20250625 07:52:17.155121 1161 ts_tablet_manager.cc:1939] T 51279fb42ae8480b8511432cb4f808ab P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.155403 1308 raft_consensus.cc:491] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.155537 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 217) took 2200 ms. Trace:
I20250625 07:52:17.155606 1308 raft_consensus.cc:513] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.156422 1308 leader_election.cc:290] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:17.157222 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59df0f66116e4dfd9386d107cab0aaf6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:17.158236 851 leader_election.cc:343] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.158495 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59df0f66116e4dfd9386d107cab0aaf6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:17.159215 850 leader_election.cc:343] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.159358 850 leader_election.cc:304] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.159875 1308 raft_consensus.cc:2747] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.163183 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_701b07f0889e4b3a8765f597fa055673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.163774 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_701b07f0889e4b3a8765f597fa055673 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.165138 1163 tablet_bootstrap.cc:654] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.165370 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.165935 1166 tablet_replica.cc:331] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.166200 1166 raft_consensus.cc:2241] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.166448 1166 raft_consensus.cc:2270] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.167253 1166 ts_tablet_manager.cc:1905] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.170950 1160 ts_tablet_manager.cc:1918] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.171272 1160 log.cc:1199] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/53c6c51a90754878977aeeaacf4773a6
W20250625 07:52:17.171504 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a33b7cab19244ec08d3821b6cc4c072d consensus state has no leader
I20250625 07:52:17.171713 1160 ts_tablet_manager.cc:1939] T 53c6c51a90754878977aeeaacf4773a6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.172039 1163 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.172168 1163 ts_tablet_manager.cc:1397] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250625 07:52:17.172853 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 53c6c51a90754878977aeeaacf4773a6 (table default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]) successfully deleted
I20250625 07:52:17.173071 1163 raft_consensus.cc:357] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.173266 1163 raft_consensus.cc:383] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.173377 1163 raft_consensus.cc:738] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.173615 1163 consensus_queue.cc:260] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.173717 1166 ts_tablet_manager.cc:1918] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.173870 1166 log.cc:1199] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/23755f70822740e08ca6a5e5b5571458
I20250625 07:52:17.174198 1163 ts_tablet_manager.cc:1428] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:17.174484 1163 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:17.176429 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a3670cadfa2841949705c65c9b6c0927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3670cadfa2841949705c65c9b6c0927 already in progress: creating tablet
I20250625 07:52:17.176761 1163 tablet_bootstrap.cc:654] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.155647 1161 rpcz_store.cc:276] 0625 07:52:14.955273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.955345 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:17.007521 (+2052176us) ts_tablet_manager.cc:1145] Deleting tablet 51279fb42ae8480b8511432cb4f808ab
0625 07:52:17.007531 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.092184 (+ 84653us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.154364 (+ 62180us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.155502 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051901}
I20250625 07:52:17.187448 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 086c9321a70e44b5bb3ea419bb466fbe from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.187606 1161 tablet_replica.cc:331] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.187747 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 51279fb42ae8480b8511432cb4f808ab (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:17.187839 1161 raft_consensus.cc:2241] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.188017 1161 raft_consensus.cc:2270] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.188640 1161 ts_tablet_manager.cc:1905] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.191960 1439 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38477 every 8 connection(s)
I20250625 07:52:17.191956 1268 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38477
I20250625 07:52:17.195137 1268 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
I20250625 07:52:17.196336 1158 tablet_bootstrap.cc:492] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.196591 1158 ts_tablet_manager.cc:1397] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.002s
W20250625 07:52:17.197181 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c50c35cac8e94e088cdacb664b8adfe1 consensus state has no leader
I20250625 07:52:17.197567 1158 raft_consensus.cc:357] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.197829 1158 raft_consensus.cc:383] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.197974 1158 raft_consensus.cc:738] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.198336 1158 consensus_queue.cc:260] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.204313 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 1268
I20250625 07:52:17.208340 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b3830bd506443ab6f06ddd3c6b9974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.209597 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fc8939aafc524fba85e19aefef0e51bc" } modify_external_catalogs: true
I20250625 07:52:17.209887 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fc8939aafc524fba85e19aefef0e51bc" } modify_external_catalogs: true
I20250625 07:52:17.210943 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56d1c8028ad54c7d8dcb1f2b342f9607" } new_table_name: "default.Table_8a130964fd79498ba4aad19cbd88c820" modify_external_catalogs: true
I20250625 07:52:17.218441 1163 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.219218 1163 ts_tablet_manager.cc:1397] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.013s
I20250625 07:52:17.220247 1163 raft_consensus.cc:357] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.210371 1158 ts_tablet_manager.cc:1428] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250625 07:52:17.221455 1163 raft_consensus.cc:383] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.221686 1163 raft_consensus.cc:738] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.222121 1163 consensus_queue.cc:260] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.222967 1163 ts_tablet_manager.cc:1428] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:17.223834 1163 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.226877 1166 ts_tablet_manager.cc:1939] T 23755f70822740e08ca6a5e5b5571458 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.190727 1162 tablet_bootstrap.cc:654] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.231520 595 master_service.cc:948] ReplaceTablet: received request to replace tablet a3670cadfa2841949705c65c9b6c0927 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.231993 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 23755f70822740e08ca6a5e5b5571458 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
I20250625 07:52:17.232602 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2cdfc224b12c489ebb46a9d55a90c823" } modify_external_catalogs: true
I20250625 07:52:17.232841 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2cdfc224b12c489ebb46a9d55a90c823" } modify_external_catalogs: true
I20250625 07:52:17.240923 1163 tablet_bootstrap.cc:654] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.242018 1158 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.245774 1162 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.246035 1162 ts_tablet_manager.cc:1397] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.095s user 0.001s sys 0.004s
I20250625 07:52:17.246802 1162 raft_consensus.cc:357] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.248073 1162 raft_consensus.cc:383] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.248205 1162 raft_consensus.cc:738] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.248534 1162 consensus_queue.cc:260] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.249287 1162 ts_tablet_manager.cc:1428] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 07:52:17.250223 1162 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.252193 1162 tablet_bootstrap.cc:654] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.257982 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 365ef91e7bac43e283602d3d036853ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.265529 1163 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.265875 1163 ts_tablet_manager.cc:1397] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20250625 07:52:17.267067 1163 raft_consensus.cc:357] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.267428 1163 raft_consensus.cc:383] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.267589 1163 raft_consensus.cc:738] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.268002 1163 consensus_queue.cc:260] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.268667 1163 ts_tablet_manager.cc:1428] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:17.270153 1163 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.270394 1162 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.270606 1162 ts_tablet_manager.cc:1397] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250625 07:52:17.272469 1163 tablet_bootstrap.cc:654] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.273437 1162 raft_consensus.cc:357] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.277225 1162 raft_consensus.cc:383] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.277412 1162 raft_consensus.cc:738] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.277760 1158 tablet_bootstrap.cc:654] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.277766 1162 consensus_queue.cc:260] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.280608 1162 ts_tablet_manager.cc:1428] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:17.281039 1162 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.281248 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_97d416631de74a18be102771d2b2a6d5" } modify_external_catalogs: true
I20250625 07:52:17.281463 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_97d416631de74a18be102771d2b2a6d5" } modify_external_catalogs: true
I20250625 07:52:17.282064 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e57f684466224dd5a4a8ff79408e1a4d" } modify_external_catalogs: true
I20250625 07:52:17.282233 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e57f684466224dd5a4a8ff79408e1a4d" } modify_external_catalogs: true
I20250625 07:52:17.283139 1162 tablet_bootstrap.cc:654] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.291381 1163 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.291618 1163 ts_tablet_manager.cc:1397] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20250625 07:52:17.293437 1163 raft_consensus.cc:357] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.293725 1163 raft_consensus.cc:383] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.293810 1163 raft_consensus.cc:738] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.293916 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.292632 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_55794188bd9d47d29bc77e1c1498512f" } modify_external_catalogs: true
I20250625 07:52:17.294149 1163 consensus_queue.cc:260] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.294342 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_55794188bd9d47d29bc77e1c1498512f" } modify_external_catalogs: true
I20250625 07:52:17.296288 1166 tablet_replica.cc:331] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.296542 1166 raft_consensus.cc:2241] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.297619 1163 ts_tablet_manager.cc:1428] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250625 07:52:17.298082 1163 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.302119 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.302819 1161 ts_tablet_manager.cc:1918] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.302980 1161 log.cc:1199] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/f774b4ac0c774ee7bf33d13a678cd368
I20250625 07:52:17.303350 1161 ts_tablet_manager.cc:1939] T f774b4ac0c774ee7bf33d13a678cd368 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.303678 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 218) took 2347 ms. Trace:
I20250625 07:52:17.303769 1161 rpcz_store.cc:276] 0625 07:52:14.956197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.956342 (+ 145us) service_pool.cc:225] Handling call
0625 07:52:17.187530 (+2231188us) ts_tablet_manager.cc:1145] Deleting tablet f774b4ac0c774ee7bf33d13a678cd368
0625 07:52:17.187544 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.247162 (+ 59618us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.302674 (+ 55512us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.303630 (+ 956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230892}
I20250625 07:52:17.304280 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet f774b4ac0c774ee7bf33d13a678cd368 (table default.Table_90101094ccf24d36a5fbb85a32358074 [id=ee25ca39f508495f80b14f5a94da7279]) successfully deleted
I20250625 07:52:17.306233 1163 tablet_bootstrap.cc:654] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.307859 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.307861 1166 raft_consensus.cc:2270] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.311062 1166 ts_tablet_manager.cc:1905] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.315796 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.323688 1166 ts_tablet_manager.cc:1918] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.323860 1166 log.cc:1199] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/365ef91e7bac43e283602d3d036853ff
I20250625 07:52:17.324095 1163 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.324307 1163 ts_tablet_manager.cc:1397] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20250625 07:52:17.325268 1163 raft_consensus.cc:357] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.325562 1163 raft_consensus.cc:383] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.325722 1163 raft_consensus.cc:738] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.326141 1163 consensus_queue.cc:260] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.330207 1166 ts_tablet_manager.cc:1939] T 365ef91e7bac43e283602d3d036853ff P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.330456 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aa09e516c85c4bc0abcb6bead66d3be1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.331072 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa09e516c85c4bc0abcb6bead66d3be1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.331395 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 365ef91e7bac43e283602d3d036853ff (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
I20250625 07:52:17.332971 1163 ts_tablet_manager.cc:1428] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:17.336956 1161 tablet_replica.cc:331] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.337172 1161 raft_consensus.cc:2241] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.337342 1161 raft_consensus.cc:2270] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.337862 1161 ts_tablet_manager.cc:1905] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.338533 1163 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:17.340360 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bfcef152c9844a59989c532bd60c07d consensus state has no leader
I20250625 07:52:17.340704 1166 tablet_replica.cc:331] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.340899 1163 tablet_bootstrap.cc:654] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.340826 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_989994a848944308835141de4c1d5a6d" } new_table_name: "default.Table_a414bf3d9fc24f58ac403f02e4994a8f" modify_external_catalogs: true
I20250625 07:52:17.340955 1166 raft_consensus.cc:2241] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.341132 1166 raft_consensus.cc:2270] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.342509 1166 ts_tablet_manager.cc:1905] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.351749 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2560f175e7af433db12e4a7c5b4c3c69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.352347 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2560f175e7af433db12e4a7c5b4c3c69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.355888 1163 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.356069 1163 ts_tablet_manager.cc:1397] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20250625 07:52:17.356859 1163 raft_consensus.cc:357] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.357054 1163 raft_consensus.cc:383] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.357113 1163 raft_consensus.cc:738] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.357364 1163 consensus_queue.cc:260] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.357887 1163 ts_tablet_manager.cc:1428] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:17.358412 1163 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.362124 1163 tablet_bootstrap.cc:654] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.363169 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 76fe692067954856908f936ced332c18 (table default.Table_527a45a533ca4babab55fa52d117496e [id=b02af454e5f6454885578ace6c10ad97]). Expected version 1 got 0
I20250625 07:52:17.365424 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.365459 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 2d8d3300befc4f419070ae84725ee96a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.365737 881 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.365972 1160 tablet_replica.cc:331] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.366238 1160 raft_consensus.cc:2241] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.366431 1160 raft_consensus.cc:2270] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.368454 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.368988 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.369459 743 tablet_service.cc:1468] Processing CreateTablet for tablet 4da889781dbd41aebac86e9833cf2b7e (DEFAULT_TABLE table=default.Table_4a0e01d2875e4b639d653e035aa60679 [id=a3fe36da128741949f4beb0021a3bcba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.369580 1166 ts_tablet_manager.cc:1918] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.369660 1160 ts_tablet_manager.cc:1905] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.369729 1166 log.cc:1199] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1eab049ad7fc4b5581e5c31a14491306
I20250625 07:52:17.369918 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da889781dbd41aebac86e9833cf2b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.370146 1166 ts_tablet_manager.cc:1939] T 1eab049ad7fc4b5581e5c31a14491306 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.374162 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1eab049ad7fc4b5581e5c31a14491306 (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
I20250625 07:52:17.375468 1373 raft_consensus.cc:491] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.375646 1373 raft_consensus.cc:513] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.376343 1373 leader_election.cc:290] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:17.377657 986 leader_election.cc:343] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:17.380591 1161 ts_tablet_manager.cc:1918] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:17.380957 985 leader_election.cc:343] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:17.381114 985 leader_election.cc:304] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:17.382477 1373 raft_consensus.cc:2747] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.383399 1160 ts_tablet_manager.cc:1918] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.385298 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 4da889781dbd41aebac86e9833cf2b7e (DEFAULT_TABLE table=default.Table_4a0e01d2875e4b639d653e035aa60679 [id=a3fe36da128741949f4beb0021a3bcba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.385825 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da889781dbd41aebac86e9833cf2b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.393455 743 tablet_service.cc:1468] Processing CreateTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e (DEFAULT_TABLE table=default.Table_dfe7357f10e546fa8c2a6bcc522beb1e [id=6fec84e2fefb4a738bce54d790e73177]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.393913 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da895c90d6ee40379fe8181c2dbd7a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.396785 1161 log.cc:1199] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6fc7c0feb0e04ef28c323323e3007efc
I20250625 07:52:17.397228 1161 ts_tablet_manager.cc:1939] T 6fc7c0feb0e04ef28c323323e3007efc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.397980 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 219) took 2440 ms. Trace:
I20250625 07:52:17.398124 1161 rpcz_store.cc:276] 0625 07:52:14.957136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.957194 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:17.336810 (+2379616us) ts_tablet_manager.cc:1145] Deleting tablet 6fc7c0feb0e04ef28c323323e3007efc
0625 07:52:17.336824 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.377220 (+ 40396us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.380469 (+ 3249us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.397941 (+ 17472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2379367}
I20250625 07:52:17.398581 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6fc7c0feb0e04ef28c323323e3007efc (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:17.384516 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4da889781dbd41aebac86e9833cf2b7e (DEFAULT_TABLE table=default.Table_4a0e01d2875e4b639d653e035aa60679 [id=a3fe36da128741949f4beb0021a3bcba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.399219 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4da889781dbd41aebac86e9833cf2b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.399695 1440 sys_catalog.cc:263] Verifying existing consensus state
I20250625 07:52:17.404315 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap starting.
I20250625 07:52:17.406543 881 tablet_service.cc:1468] Processing CreateTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e (DEFAULT_TABLE table=default.Table_dfe7357f10e546fa8c2a6bcc522beb1e [id=6fec84e2fefb4a738bce54d790e73177]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.406817 1014 tablet_service.cc:1468] Processing CreateTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e (DEFAULT_TABLE table=default.Table_dfe7357f10e546fa8c2a6bcc522beb1e [id=6fec84e2fefb4a738bce54d790e73177]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.406849 750 tablet_service.cc:1468] Processing CreateTablet for tablet 173a565916c1440493ece348599ddba3 (DEFAULT_TABLE table=default.Table_82cd9b0cdb87402c90df8c979e56570a [id=b1ec5aff86df4e9381d955d406326bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.407063 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da895c90d6ee40379fe8181c2dbd7a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.407208 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173a565916c1440493ece348599ddba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.407238 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da895c90d6ee40379fe8181c2dbd7a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.412905 1373 raft_consensus.cc:491] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.413108 1373 raft_consensus.cc:513] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.413764 1373 leader_election.cc:290] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:17.414819 985 leader_election.cc:343] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2b020ff9f04b4d1796af4ff585cc04aa
W20250625 07:52:17.414971 986 leader_election.cc:343] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2b020ff9f04b4d1796af4ff585cc04aa
I20250625 07:52:17.415156 986 leader_election.cc:304] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:17.415719 1373 raft_consensus.cc:2747] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.420032 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5b4da047314e43ebbb8920e13911845c" } new_table_name: "default.Table_35e64151eae847128b84b16f6c0ab3ea" modify_external_catalogs: true
I20250625 07:52:17.420652 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 220) took 2461 ms. Trace:
I20250625 07:52:17.420805 1161 rpcz_store.cc:276] 0625 07:52:14.959437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.959484 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:17.420499 (+2461015us) ts_tablet_manager.cc:1145] Deleting tablet 445bfed532004108a12c8fd78370c1cb
0625 07:52:17.420511 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.420616 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460815}
I20250625 07:52:17.420929 1160 log.cc:1199] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9c770b41e0844786ad81c7b3579f9934
W20250625 07:52:17.421298 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 445bfed532004108a12c8fd78370c1cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 445bfed532004108a12c8fd78370c1cb already in progress: creating tablet
I20250625 07:52:17.422024 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 221) took 2460 ms. Trace:
I20250625 07:52:17.422155 1161 rpcz_store.cc:276] 0625 07:52:14.961064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.961110 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:17.421895 (+2460785us) ts_tablet_manager.cc:1145] Deleting tablet 651c74bf9a9b47d6ab5a875a5c304783
0625 07:52:17.421905 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.421989 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460593}
W20250625 07:52:17.422595 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 651c74bf9a9b47d6ab5a875a5c304783 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 651c74bf9a9b47d6ab5a875a5c304783 already in progress: creating tablet
I20250625 07:52:17.423662 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 173a565916c1440493ece348599ddba3 (DEFAULT_TABLE table=default.Table_82cd9b0cdb87402c90df8c979e56570a [id=b1ec5aff86df4e9381d955d406326bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.423684 750 tablet_service.cc:1468] Processing CreateTablet for tablet 896419bd407f43e1a0a361330ff9821b (DEFAULT_TABLE table=default.Table_406f7f4727764ec1be3f67c739f08a6d [id=3a39e84d0aa349689ccc22a31ea282b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.424073 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896419bd407f43e1a0a361330ff9821b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.424098 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173a565916c1440493ece348599ddba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.424208 881 tablet_service.cc:1468] Processing CreateTablet for tablet 173a565916c1440493ece348599ddba3 (DEFAULT_TABLE table=default.Table_82cd9b0cdb87402c90df8c979e56570a [id=b1ec5aff86df4e9381d955d406326bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.424921 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 173a565916c1440493ece348599ddba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.428965 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 222) took 2461 ms. Trace:
I20250625 07:52:17.424532 1160 ts_tablet_manager.cc:1939] T 9c770b41e0844786ad81c7b3579f9934 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.429096 1161 rpcz_store.cc:276] 0625 07:52:14.967699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.967754 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:17.428814 (+2461060us) ts_tablet_manager.cc:1145] Deleting tablet e7100ef72ff14c24be92c21c6eb8a9de
0625 07:52:17.428824 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.428929 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2460846}
I20250625 07:52:17.429708 1161 tablet_replica.cc:331] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.429908 1161 raft_consensus.cc:2241] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.430076 1161 raft_consensus.cc:2270] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.430940 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9c770b41e0844786ad81c7b3579f9934 (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:17.433099 1161 ts_tablet_manager.cc:1905] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.433470 1160 tablet_replica.cc:331] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.434330 1160 raft_consensus.cc:2241] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.434636 1160 raft_consensus.cc:2270] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.433945 890 tablet_service.cc:1468] Processing CreateTablet for tablet 896419bd407f43e1a0a361330ff9821b (DEFAULT_TABLE table=default.Table_406f7f4727764ec1be3f67c739f08a6d [id=3a39e84d0aa349689ccc22a31ea282b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.435303 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896419bd407f43e1a0a361330ff9821b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.438577 1160 ts_tablet_manager.cc:1905] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.450325 1158 tablet_bootstrap.cc:492] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.450800 1158 ts_tablet_manager.cc:1397] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.209s user 0.001s sys 0.004s
I20250625 07:52:17.451659 1158 raft_consensus.cc:357] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.451876 1158 raft_consensus.cc:383] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.451956 1158 raft_consensus.cc:738] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.452293 1158 consensus_queue.cc:260] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.454391 1158 ts_tablet_manager.cc:1428] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:17.454738 1158 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.457096 747 tablet_service.cc:1468] Processing CreateTablet for tablet 5179a144a9a0445f933ca02e79bbc122 (DEFAULT_TABLE table=default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.457543 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5179a144a9a0445f933ca02e79bbc122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.458321 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 086c9321a70e44b5bb3ea419bb466fbe
I20250625 07:52:17.458828 590 catalog_manager.cc:6652] ReplaceTablet: tablet 086c9321a70e44b5bb3ea419bb466fbe deleted and replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516
I20250625 07:52:17.459920 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.460511 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cb6a73fc11c44f4ead125864cc91710d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.461263 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.461658 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 3338d1c708b4421c8501740da89afec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:17.462246 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb6a73fc11c44f4ead125864cc91710d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:17.462942 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 086c9321a70e44b5bb3ea419bb466fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c9321a70e44b5bb3ea419bb466fbe already in progress: creating tablet
W20250625 07:52:17.461495 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a630f8804bbf41de86b08cde419643ff consensus state has no leader
I20250625 07:52:17.465059 596 master_service.cc:948] ReplaceTablet: received request to replace tablet d92962b9e7d14c8d88d775b98e75de35 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.466331 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_406f7f4727764ec1be3f67c739f08a6d" } new_table_name: "default.Table_3c91554bb23646638781392298543d03" modify_external_catalogs: true
I20250625 07:52:17.467919 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 896419bd407f43e1a0a361330ff9821b (DEFAULT_TABLE table=default.Table_406f7f4727764ec1be3f67c739f08a6d [id=3a39e84d0aa349689ccc22a31ea282b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.468400 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896419bd407f43e1a0a361330ff9821b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.472936 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.472956 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 5179a144a9a0445f933ca02e79bbc122 (DEFAULT_TABLE table=default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.473459 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5179a144a9a0445f933ca02e79bbc122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.473481 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc8f47541c54c4287ebb7b892daa1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.477289 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.477773 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc8f47541c54c4287ebb7b892daa1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.482887 1013 tablet_service.cc:1468] Processing CreateTablet for tablet a41ffd566593499299aba5513ccb2f30 (DEFAULT_TABLE table=default.Table_d7220ef926924a20949121c91fa08801 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.483374 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ffd566593499299aba5513ccb2f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.487337 1012 tablet_service.cc:1468] Processing CreateTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e (DEFAULT_TABLE table=default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:17.487933 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fcc3afc210a540058294121d2b83c66d (table default.Table_b167db2ea8944d13b34adbb44e815b62 [id=8a01186fa596413fb75529d0d57a8d9d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet fcc3afc210a540058294121d2b83c66d consensus state has no leader
I20250625 07:52:17.488317 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d8b6b5c06248879e6052f6ae7fe46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.492795 757 tablet_service.cc:1468] Processing CreateTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e (DEFAULT_TABLE table=default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.493306 756 tablet_service.cc:1468] Processing CreateTablet for tablet a41ffd566593499299aba5513ccb2f30 (DEFAULT_TABLE table=default.Table_d7220ef926924a20949121c91fa08801 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.494051 744 tablet_service.cc:1468] Processing CreateTablet for tablet 9f1d251171314118b66e3a921a4a50c5 (DEFAULT_TABLE table=default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.500385 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d8b6b5c06248879e6052f6ae7fe46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.502058 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ffd566593499299aba5513ccb2f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.504307 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 110fe299c7c44df69f459da72a1138a5
I20250625 07:52:17.505622 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 743123ffcf8240dc82bea7178559b7ea from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.506209 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f1d251171314118b66e3a921a4a50c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.508632 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 541c983d64f348539979f74199f79e59
I20250625 07:52:17.509560 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_268b3ed7299240f3abb976c1f05dfda6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.510258 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_268b3ed7299240f3abb976c1f05dfda6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.512463 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3b9491eb269948a29888a7301ea75d42
I20250625 07:52:17.513319 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_527a45a533ca4babab55fa52d117496e" } new_table_name: "default.Table_33aedac05c3649c0a21c0367c4ff59a3" modify_external_catalogs: true
I20250625 07:52:17.513645 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.514010 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.514374 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3b9491eb269948a29888a7301ea75d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b9491eb269948a29888a7301ea75d42 already in progress: creating tablet
W20250625 07:52:17.514537 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.516981 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.517194 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.518898 1158 tablet_bootstrap.cc:654] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.520197 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9f1d251171314118b66e3a921a4a50c5 (DEFAULT_TABLE table=default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.520627 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f1d251171314118b66e3a921a4a50c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.522125 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9bccda893fd5466886a4d253172309e1 (DEFAULT_TABLE table=default.Table_f16f5e9c3f6d4110baa383f8551d2f91 [id=c90351757d464813b5d0d67936dd69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.522529 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bccda893fd5466886a4d253172309e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.523216 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.531709 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.532459 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.532478 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 110fe299c7c44df69f459da72a1138a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110fe299c7c44df69f459da72a1138a5 already in progress: creating tablet
W20250625 07:52:17.533088 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 541c983d64f348539979f74199f79e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541c983d64f348539979f74199f79e59 already in progress: creating tablet
I20250625 07:52:17.548472 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.549103 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a90bb52358f4c38a540ad025e9e0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.553184 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:17.553783 595 catalog_manager.cc:6652] ReplaceTablet: tablet 2d8d3300befc4f419070ae84725ee96a deleted and replaced by tablet 3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:17.554349 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.554975 1161 ts_tablet_manager.cc:1918] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.555124 1161 log.cc:1199] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/5de10cbe72c8477ebe83831c869c3df1
I20250625 07:52:17.555469 1161 ts_tablet_manager.cc:1939] T 5de10cbe72c8477ebe83831c869c3df1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.557000 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9bccda893fd5466886a4d253172309e1 (DEFAULT_TABLE table=default.Table_f16f5e9c3f6d4110baa383f8551d2f91 [id=c90351757d464813b5d0d67936dd69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.557488 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bccda893fd5466886a4d253172309e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.557557 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 223) took 2589 ms. Trace:
I20250625 07:52:17.557651 1161 rpcz_store.cc:276] 0625 07:52:14.968336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.968384 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:17.429652 (+2461268us) ts_tablet_manager.cc:1145] Deleting tablet 5de10cbe72c8477ebe83831c869c3df1
0625 07:52:17.429660 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.481930 (+ 52270us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.554865 (+ 72935us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.557505 (+ 2640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2461106}
I20250625 07:52:17.558121 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 226) took 2559 ms. Trace:
I20250625 07:52:17.558182 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 5de10cbe72c8477ebe83831c869c3df1 (table default.Table_1af8e3ae3fb04f159b97af62f5256fa8 [id=b8805d5a925e4d4d99547ca8be1eff59]) successfully deleted
I20250625 07:52:17.558230 1161 rpcz_store.cc:276] 0625 07:52:14.998217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:14.998298 (+ 81us) service_pool.cc:225] Handling call
0625 07:52:17.557947 (+2559649us) ts_tablet_manager.cc:1145] Deleting tablet 3ca0144e3c5142ac9fcbd927b7585a3c
0625 07:52:17.557957 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.558073 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2559435}
I20250625 07:52:17.558634 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 228) took 2553 ms. Trace:
W20250625 07:52:17.558701 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3ca0144e3c5142ac9fcbd927b7585a3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ca0144e3c5142ac9fcbd927b7585a3c already in progress: creating tablet
I20250625 07:52:17.558727 1161 rpcz_store.cc:276] 0625 07:52:15.005524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.005591 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:17.558489 (+2552898us) ts_tablet_manager.cc:1145] Deleting tablet 7ad78944352e4274acd8d9889b1d32b8
0625 07:52:17.558500 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.558587 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2552632}
I20250625 07:52:17.559113 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 229) took 2551 ms. Trace:
W20250625 07:52:17.559252 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7ad78944352e4274acd8d9889b1d32b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ad78944352e4274acd8d9889b1d32b8 already in progress: creating tablet
I20250625 07:52:17.559202 1161 rpcz_store.cc:276] 0625 07:52:15.007313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.007378 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:17.558959 (+2551581us) ts_tablet_manager.cc:1145] Deleting tablet d92962b9e7d14c8d88d775b98e75de35
0625 07:52:17.558967 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.559060 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2551337}
I20250625 07:52:17.559587 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 230) took 2549 ms. Trace:
W20250625 07:52:17.559692 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d92962b9e7d14c8d88d775b98e75de35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d92962b9e7d14c8d88d775b98e75de35 already in progress: creating tablet
I20250625 07:52:17.559670 1161 rpcz_store.cc:276] 0625 07:52:15.009887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.009944 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:17.559433 (+2549489us) ts_tablet_manager.cc:1145] Deleting tablet 7f196473315441c6bf8b5173ed46f833
0625 07:52:17.559441 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.559536 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2549258}
W20250625 07:52:17.560125 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7f196473315441c6bf8b5173ed46f833 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f196473315441c6bf8b5173ed46f833 already in progress: creating tablet
I20250625 07:52:17.563992 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.564607 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2d8d3300befc4f419070ae84725ee96a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8d3300befc4f419070ae84725ee96a already in progress: creating tablet
I20250625 07:52:17.565659 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.566088 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a90bb52358f4c38a540ad025e9e0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.570648 1308 raft_consensus.cc:491] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.572475 1158 tablet_bootstrap.cc:492] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.572635 1158 ts_tablet_manager.cc:1397] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.118s user 0.000s sys 0.004s
I20250625 07:52:17.573487 1158 raft_consensus.cc:357] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.573665 1158 raft_consensus.cc:383] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.573751 1158 raft_consensus.cc:738] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.574019 1158 consensus_queue.cc:260] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.574451 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:17.575515 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:17.575729 1158 ts_tablet_manager.cc:1428] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:17.575949 747 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.576335 745 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.576426 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 875fa1b46903452eaed1f2235d63dfdf consensus state has no leader
W20250625 07:52:17.576589 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ba0735d576f947d395bee8d1b865f2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba0735d576f947d395bee8d1b865f2fd already in progress: creating tablet
W20250625 07:52:17.576929 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 already in progress: creating tablet
I20250625 07:52:17.578547 1308 raft_consensus.cc:513] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.578819 1161 tablet_replica.cc:331] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.578955 1373 raft_consensus.cc:491] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.578997 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.579046 1161 raft_consensus.cc:2241] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.579133 1373 raft_consensus.cc:513] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.579259 1308 leader_election.cc:290] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:17.579694 1161 raft_consensus.cc:2270] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.579722 1373 leader_election.cc:290] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:17.579829 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:17.580091 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.580246 1161 ts_tablet_manager.cc:1905] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:17.583390 986 leader_election.cc:343] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: d51a898202154881ab4ad41da1d519a3
I20250625 07:52:17.585091 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.588140 895 tablet_service.cc:1468] Processing CreateTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.588580 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc8f47541c54c4287ebb7b892daa1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.589107 881 tablet_service.cc:1468] Processing CreateTablet for tablet a41ffd566593499299aba5513ccb2f30 (DEFAULT_TABLE table=default.Table_d7220ef926924a20949121c91fa08801 [id=16459cf0283447e9a445cf210eaa88f5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:17.589154 985 leader_election.cc:343] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d51a898202154881ab4ad41da1d519a3
I20250625 07:52:17.589291 985 leader_election.cc:304] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:17.589555 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a41ffd566593499299aba5513ccb2f30. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:17.590023 851 leader_election.cc:343] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.590070 1373 raft_consensus.cc:2747] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.590175 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5179a144a9a0445f933ca02e79bbc122 (DEFAULT_TABLE table=default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.590880 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5179a144a9a0445f933ca02e79bbc122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.591351 1162 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.592142 1158 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.592381 1162 ts_tablet_manager.cc:1397] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.311s user 0.039s sys 0.085s
I20250625 07:52:17.593453 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.593657 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.593976 1158 tablet_bootstrap.cc:654] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.595798 885 tablet_service.cc:1468] Processing CreateTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e (DEFAULT_TABLE table=default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.596240 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d8b6b5c06248879e6052f6ae7fe46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.596819 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9f1d251171314118b66e3a921a4a50c5 (DEFAULT_TABLE table=default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.597374 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f1d251171314118b66e3a921a4a50c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.597352 1162 raft_consensus.cc:357] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.598275 1162 raft_consensus.cc:383] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.599056 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.598807 1162 raft_consensus.cc:738] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.599215 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.599745 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a90bb52358f4c38a540ad025e9e0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.599895 1162 consensus_queue.cc:260] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.600453 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9bccda893fd5466886a4d253172309e1 (DEFAULT_TABLE table=default.Table_f16f5e9c3f6d4110baa383f8551d2f91 [id=c90351757d464813b5d0d67936dd69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.600984 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bccda893fd5466886a4d253172309e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.602428 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.602605 1440 log.cc:826] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:17.602650 882 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.602799 1162 ts_tablet_manager.cc:1428] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250625 07:52:17.603766 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:17.604135 1162 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.604321 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:17.605448 850 leader_election.cc:343] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.605592 850 leader_election.cc:304] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.606122 1308 raft_consensus.cc:2747] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.612735 1162 tablet_bootstrap.cc:654] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.614743 1161 ts_tablet_manager.cc:1918] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.614924 1161 log.cc:1199] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a558e0a5403d4489892dfce9587e090c
I20250625 07:52:17.616804 1161 ts_tablet_manager.cc:1939] T a558e0a5403d4489892dfce9587e090c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.617188 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 248) took 2405 ms. Trace:
I20250625 07:52:17.617311 1161 rpcz_store.cc:276] 0625 07:52:15.211995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.212088 (+ 93us) service_pool.cc:225] Handling call
0625 07:52:17.559899 (+2347811us) ts_tablet_manager.cc:1145] Deleting tablet a558e0a5403d4489892dfce9587e090c
0625 07:52:17.559908 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.600641 (+ 40733us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.614587 (+ 13946us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.617151 (+ 2564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346097}
I20250625 07:52:17.617816 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a558e0a5403d4489892dfce9587e090c (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
I20250625 07:52:17.620934 1161 tablet_replica.cc:331] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.621248 1161 raft_consensus.cc:2241] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.621479 1161 raft_consensus.cc:2270] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.622260 1161 ts_tablet_manager.cc:1905] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.628643 1162 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.629383 1162 ts_tablet_manager.cc:1397] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20250625 07:52:17.633380 1161 ts_tablet_manager.cc:1918] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.633564 1161 log.cc:1199] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b09dc0b8a496402b82f98daed9af0d12
I20250625 07:52:17.633145 1162 raft_consensus.cc:357] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.633927 1161 ts_tablet_manager.cc:1939] T b09dc0b8a496402b82f98daed9af0d12 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.634279 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 247) took 2432 ms. Trace:
I20250625 07:52:17.634370 1161 rpcz_store.cc:276] 0625 07:52:15.201754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.203568 (+ 1814us) service_pool.cc:225] Handling call
0625 07:52:17.620838 (+2417270us) ts_tablet_manager.cc:1145] Deleting tablet b09dc0b8a496402b82f98daed9af0d12
0625 07:52:17.620855 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.628874 (+ 8019us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.633235 (+ 4361us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.634227 (+ 992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2339926}
I20250625 07:52:17.634289 1162 raft_consensus.cc:383] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.634897 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b09dc0b8a496402b82f98daed9af0d12 (table default.Table_c931105d39a449fdbdc73dc16830a6a4 [id=7f77bfd2215a491d8fba698477109409]) successfully deleted
I20250625 07:52:17.634824 1162 raft_consensus.cc:738] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.635429 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 252) took 2362 ms. Trace:
I20250625 07:52:17.635532 1161 rpcz_store.cc:276] 0625 07:52:15.272838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.272889 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:17.635239 (+2362350us) ts_tablet_manager.cc:1145] Deleting tablet a5a3751333bc48fc8a59b0e7b67a9935
0625 07:52:17.635248 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.635380 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2353328}
I20250625 07:52:17.636298 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 249) took 2398 ms. Trace:
I20250625 07:52:17.635746 1162 consensus_queue.cc:260] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.636387 1161 rpcz_store.cc:276] 0625 07:52:15.237395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.237512 (+ 117us) service_pool.cc:225] Handling call
0625 07:52:17.636160 (+2398648us) ts_tablet_manager.cc:1145] Deleting tablet 95a763abda074d358f176fd7e7be0e5c
0625 07:52:17.636170 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.636249 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2350718}
W20250625 07:52:17.636788 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 95a763abda074d358f176fd7e7be0e5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95a763abda074d358f176fd7e7be0e5c already in progress: creating tablet
I20250625 07:52:17.638622 1162 ts_tablet_manager.cc:1428] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 07:52:17.638976 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.639966 1162 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.642612 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 250) took 2401 ms. Trace:
I20250625 07:52:17.642745 1161 rpcz_store.cc:276] 0625 07:52:15.241548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.242001 (+ 453us) service_pool.cc:225] Handling call
0625 07:52:17.636766 (+2394765us) ts_tablet_manager.cc:1145] Deleting tablet 802c22a6346442c88abbeb4b9ef20c57
0625 07:52:17.636774 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.642528 (+ 5754us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3146,"tablet-delete.queue_time_us":2349098}
W20250625 07:52:17.643296 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 802c22a6346442c88abbeb4b9ef20c57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 802c22a6346442c88abbeb4b9ef20c57 already in progress: creating tablet
I20250625 07:52:17.645197 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 251) took 2372 ms. Trace:
I20250625 07:52:17.645309 1161 rpcz_store.cc:276] 0625 07:52:15.272639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.275006 (+ 2367us) service_pool.cc:225] Handling call
0625 07:52:17.644978 (+2369972us) ts_tablet_manager.cc:1145] Deleting tablet a7cb2bcdd07c42e680eb4f155212423f
0625 07:52:17.644992 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.645131 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2699,"tablet-delete.queue_time_us":2356751}
I20250625 07:52:17.646099 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 255) took 2372 ms. Trace:
I20250625 07:52:17.646205 1161 rpcz_store.cc:276] 0625 07:52:15.273315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.290844 (+ 17529us) service_pool.cc:225] Handling call
0625 07:52:17.645894 (+2355050us) ts_tablet_manager.cc:1145] Deleting tablet 2db9e8d0fefa47bfaeee462dba096b49
0625 07:52:17.645908 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.646005 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2340194}
W20250625 07:52:17.646719 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2db9e8d0fefa47bfaeee462dba096b49 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2db9e8d0fefa47bfaeee462dba096b49 already in progress: creating tablet
I20250625 07:52:17.647168 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 256) took 2373 ms. Trace:
I20250625 07:52:17.647274 1161 rpcz_store.cc:276] 0625 07:52:15.273435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.290944 (+ 17509us) service_pool.cc:225] Handling call
0625 07:52:17.646983 (+2356039us) ts_tablet_manager.cc:1145] Deleting tablet 7db110bb6cc146a393d62ddabbe1955a
0625 07:52:17.646995 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.647103 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2341088}
W20250625 07:52:17.647722 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7db110bb6cc146a393d62ddabbe1955a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db110bb6cc146a393d62ddabbe1955a already in progress: creating tablet
I20250625 07:52:17.648088 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 254) took 2374 ms. Trace:
I20250625 07:52:17.648181 1161 rpcz_store.cc:276] 0625 07:52:15.273079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.290772 (+ 17693us) service_pool.cc:225] Handling call
0625 07:52:17.647949 (+2357177us) ts_tablet_manager.cc:1145] Deleting tablet 9c770b41e0844786ad81c7b3579f9934
0625 07:52:17.647957 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.648028 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2337650}
I20250625 07:52:17.648891 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 264) took 2374 ms. Trace:
I20250625 07:52:17.649003 1161 rpcz_store.cc:276] 0625 07:52:15.274179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.351135 (+ 76956us) service_pool.cc:225] Handling call
0625 07:52:17.648656 (+2297521us) ts_tablet_manager.cc:1145] Deleting tablet f9a94dc1432d4cbdbf3eecacb1429c23
0625 07:52:17.648664 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.648820 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2279080}
W20250625 07:52:17.649497 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f9a94dc1432d4cbdbf3eecacb1429c23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9a94dc1432d4cbdbf3eecacb1429c23 already in progress: creating tablet
I20250625 07:52:17.649616 1162 tablet_bootstrap.cc:654] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.649766 1161 tablet_replica.cc:331] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.649971 1161 raft_consensus.cc:2241] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.650163 1161 raft_consensus.cc:2270] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.650682 1161 ts_tablet_manager.cc:1905] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.658218 1161 ts_tablet_manager.cc:1918] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.658373 1161 log.cc:1199] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/55d190e9d11c4589a82fcc7307e9eccb
I20250625 07:52:17.658915 1161 ts_tablet_manager.cc:1939] T 55d190e9d11c4589a82fcc7307e9eccb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.659312 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 266) took 2384 ms. Trace:
I20250625 07:52:17.659431 1161 rpcz_store.cc:276] 0625 07:52:15.274363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.377739 (+103376us) service_pool.cc:225] Handling call
0625 07:52:17.649690 (+2271951us) ts_tablet_manager.cc:1145] Deleting tablet 55d190e9d11c4589a82fcc7307e9eccb
0625 07:52:17.649700 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.653751 (+ 4051us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.657797 (+ 4046us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.659270 (+ 1473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266417}
I20250625 07:52:17.659830 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 55d190e9d11c4589a82fcc7307e9eccb (table default.Table_2d6ea477268d47f58b30102e5dc88635 [id=516fdfcb90fd42d690397c77e99b4a38]) successfully deleted
I20250625 07:52:17.660382 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 267) took 2385 ms. Trace:
I20250625 07:52:17.660537 1161 rpcz_store.cc:276] 0625 07:52:15.274443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.383304 (+108861us) service_pool.cc:225] Handling call
0625 07:52:17.660258 (+2276954us) ts_tablet_manager.cc:1145] Deleting tablet 366d3e6fcc9d49be8fbd2cd070632861
0625 07:52:17.660268 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.660352 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2274419}
W20250625 07:52:17.661077 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 366d3e6fcc9d49be8fbd2cd070632861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 366d3e6fcc9d49be8fbd2cd070632861 already in progress: creating tablet
I20250625 07:52:17.663398 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 268) took 2388 ms. Trace:
I20250625 07:52:17.664755 1162 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.665464 1162 ts_tablet_manager.cc:1397] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20250625 07:52:17.669857 1162 raft_consensus.cc:357] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.670797 1162 raft_consensus.cc:383] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.668866 1373 raft_consensus.cc:491] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.671092 1162 raft_consensus.cc:738] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.671587 1373 raft_consensus.cc:513] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.672448 1373 leader_election.cc:290] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:17.672319 1162 consensus_queue.cc:260] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.674998 1162 ts_tablet_manager.cc:1428] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.003s sys 0.001s
I20250625 07:52:17.676362 1162 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:17.677513 985 leader_election.cc:343] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 09b3830bd506443ab6f06ddd3c6b9974
W20250625 07:52:17.678416 986 leader_election.cc:343] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 09b3830bd506443ab6f06ddd3c6b9974
I20250625 07:52:17.678531 986 leader_election.cc:304] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:17.681114 1373 raft_consensus.cc:2747] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.684818 1162 tablet_bootstrap.cc:654] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.663561 1161 rpcz_store.cc:276] 0625 07:52:15.274530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.385092 (+110562us) service_pool.cc:225] Handling call
0625 07:52:17.663267 (+2278175us) ts_tablet_manager.cc:1145] Deleting tablet a0518fbc7f52454296ae79a3c92ef800
0625 07:52:17.663277 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.663362 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262322}
W20250625 07:52:17.686465 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a0518fbc7f52454296ae79a3c92ef800 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0518fbc7f52454296ae79a3c92ef800 already in progress: creating tablet
I20250625 07:52:17.687054 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 285) took 1900 ms. Trace:
I20250625 07:52:17.687212 1161 rpcz_store.cc:276] 0625 07:52:15.786194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.786649 (+ 455us) service_pool.cc:225] Handling call
0625 07:52:17.686886 (+1900237us) ts_tablet_manager.cc:1145] Deleting tablet 494cacf2dcbb40d3b2c9f05f64809870
0625 07:52:17.686898 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.687013 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898229}
W20250625 07:52:17.688270 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 494cacf2dcbb40d3b2c9f05f64809870 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 494cacf2dcbb40d3b2c9f05f64809870 already in progress: creating tablet
I20250625 07:52:17.688951 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 284) took 1902 ms. Trace:
I20250625 07:52:17.689546 1161 rpcz_store.cc:276] 0625 07:52:15.786111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.786612 (+ 501us) service_pool.cc:225] Handling call
0625 07:52:17.688664 (+1902052us) ts_tablet_manager.cc:1145] Deleting tablet 3ab38e0cdcd14f209e6877f2e4a46e98
0625 07:52:17.688674 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.688902 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1889013}
I20250625 07:52:17.689908 1163 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.690090 1163 ts_tablet_manager.cc:1397] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.332s user 0.042s sys 0.062s
I20250625 07:52:17.691080 1163 raft_consensus.cc:357] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.691296 1163 raft_consensus.cc:383] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.691399 1163 raft_consensus.cc:738] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.691681 1163 consensus_queue.cc:260] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:17.691761 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ab38e0cdcd14f209e6877f2e4a46e98 already in progress: creating tablet
I20250625 07:52:17.692317 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 282) took 1906 ms. Trace:
I20250625 07:52:17.692507 1161 rpcz_store.cc:276] 0625 07:52:15.785567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.823448 (+ 37881us) service_pool.cc:225] Handling call
0625 07:52:17.692148 (+1868700us) ts_tablet_manager.cc:1145] Deleting tablet 2771159858fc41a1ad930229986edd92
0625 07:52:17.692162 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.692270 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862459}
I20250625 07:52:17.697165 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.701910 1160 ts_tablet_manager.cc:1918] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.702138 1160 log.cc:1199] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d92962b9e7d14c8d88d775b98e75de35
I20250625 07:52:17.702605 1160 ts_tablet_manager.cc:1939] T d92962b9e7d14c8d88d775b98e75de35 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.702926 1162 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.703576 1162 ts_tablet_manager.cc:1397] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20250625 07:52:17.703872 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d92962b9e7d14c8d88d775b98e75de35 (table default.Table_69159065e6254990b2bebfab94d2fd0b [id=eecbc272d0844eaaaa51584b812f2321]) successfully deleted
I20250625 07:52:17.704854 1160 tablet_replica.cc:331] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:17.705174 1160 raft_consensus.cc:2241] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:17.708518 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2771159858fc41a1ad930229986edd92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2771159858fc41a1ad930229986edd92 already in progress: creating tablet
I20250625 07:52:17.709302 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 283) took 1923 ms. Trace:
I20250625 07:52:17.708525 1162 raft_consensus.cc:357] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.709605 1161 rpcz_store.cc:276] 0625 07:52:15.786020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.840775 (+ 54755us) service_pool.cc:225] Handling call
0625 07:52:17.709057 (+1868282us) ts_tablet_manager.cc:1145] Deleting tablet 2a9945f38ef148caac59acaadcb2ef4c
0625 07:52:17.709074 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.709241 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867771}
I20250625 07:52:17.709702 1162 raft_consensus.cc:383] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.710116 1162 raft_consensus.cc:738] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:17.710353 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2a9945f38ef148caac59acaadcb2ef4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a9945f38ef148caac59acaadcb2ef4c already in progress: creating tablet
I20250625 07:52:17.710904 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 286) took 1838 ms. Trace:
I20250625 07:52:17.711112 1161 rpcz_store.cc:276] 0625 07:52:15.872294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:15.876416 (+ 4122us) service_pool.cc:225] Handling call
0625 07:52:17.710723 (+1834307us) ts_tablet_manager.cc:1145] Deleting tablet c81396ee6ed7400baa30dcc5d71ba45d
0625 07:52:17.710737 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.710850 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834066}
W20250625 07:52:17.711687 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c81396ee6ed7400baa30dcc5d71ba45d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c81396ee6ed7400baa30dcc5d71ba45d already in progress: creating tablet
I20250625 07:52:17.711073 1162 consensus_queue.cc:260] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.712244 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 293) took 1709 ms. Trace:
I20250625 07:52:17.713613 1161 rpcz_store.cc:276] 0625 07:52:16.002743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.002814 (+ 71us) service_pool.cc:225] Handling call
0625 07:52:17.712071 (+1709257us) ts_tablet_manager.cc:1145] Deleting tablet 0f566cc9cdd447798a5cb408ef9cd35a
0625 07:52:17.712084 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.712195 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1708909}
W20250625 07:52:17.714228 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0f566cc9cdd447798a5cb408ef9cd35a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f566cc9cdd447798a5cb408ef9cd35a already in progress: creating tablet
I20250625 07:52:17.714643 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 304) took 1471 ms. Trace:
I20250625 07:52:17.714911 1161 rpcz_store.cc:276] 0625 07:52:16.242835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.242889 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:17.714517 (+1471628us) ts_tablet_manager.cc:1145] Deleting tablet 3c8c317c10e14332a3b3919808e4a512
0625 07:52:17.714526 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.714607 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471371}
W20250625 07:52:17.715377 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3c8c317c10e14332a3b3919808e4a512 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c8c317c10e14332a3b3919808e4a512 already in progress: creating tablet
I20250625 07:52:17.716173 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 305) took 1462 ms. Trace:
I20250625 07:52:17.716377 1161 rpcz_store.cc:276] 0625 07:52:16.254167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.254287 (+ 120us) service_pool.cc:225] Handling call
0625 07:52:17.715686 (+1461399us) ts_tablet_manager.cc:1145] Deleting tablet 365ef91e7bac43e283602d3d036853ff
0625 07:52:17.715699 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.716129 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451230}
W20250625 07:52:17.717020 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 365ef91e7bac43e283602d3d036853ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 365ef91e7bac43e283602d3d036853ff already in progress: creating tablet
I20250625 07:52:17.716940 1162 ts_tablet_manager.cc:1428] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250625 07:52:17.718600 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 306) took 1456 ms. Trace:
I20250625 07:52:17.718775 1161 rpcz_store.cc:276] 0625 07:52:16.262643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.262751 (+ 108us) service_pool.cc:225] Handling call
0625 07:52:17.718470 (+1455719us) ts_tablet_manager.cc:1145] Deleting tablet 450727d8f33e4eb489969b7974cdeeb1
0625 07:52:17.718479 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.718564 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450859}
W20250625 07:52:17.719244 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 450727d8f33e4eb489969b7974cdeeb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 450727d8f33e4eb489969b7974cdeeb1 already in progress: creating tablet
I20250625 07:52:17.719880 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 308) took 1447 ms. Trace:
I20250625 07:52:17.720049 1161 rpcz_store.cc:276] 0625 07:52:16.272052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.272148 (+ 96us) service_pool.cc:225] Handling call
0625 07:52:17.719750 (+1447602us) ts_tablet_manager.cc:1145] Deleting tablet 4384bc544af149338514d40adf683f34
0625 07:52:17.719759 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.719840 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442118}
W20250625 07:52:17.721156 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4384bc544af149338514d40adf683f34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4384bc544af149338514d40adf683f34 already in progress: creating tablet
I20250625 07:52:17.721693 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 307) took 1455 ms. Trace:
I20250625 07:52:17.721858 1161 rpcz_store.cc:276] 0625 07:52:16.268259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.268441 (+ 182us) service_pool.cc:225] Handling call
0625 07:52:17.721563 (+1453122us) ts_tablet_manager.cc:1145] Deleting tablet a777f64425384be5bccc86b197626618
0625 07:52:17.721572 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.721655 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441954}
I20250625 07:52:17.722585 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 313) took 1441 ms. Trace:
I20250625 07:52:17.722755 1161 rpcz_store.cc:276] 0625 07:52:16.281295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.281338 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:17.722418 (+1441080us) ts_tablet_manager.cc:1145] Deleting tablet 31942f0998bb4118a91f476325851d5c
0625 07:52:17.722426 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.722549 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438967}
W20250625 07:52:17.723237 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 31942f0998bb4118a91f476325851d5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31942f0998bb4118a91f476325851d5c already in progress: creating tablet
I20250625 07:52:17.723564 1161 tablet_replica.cc:331] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.723814 1161 raft_consensus.cc:2241] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.724066 1161 raft_consensus.cc:2270] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.724397 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8c340466a42e47dead226f74c8176f49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.724931 1161 ts_tablet_manager.cc:1905] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.725405 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1c370799a55b474080d93cb980cf5f39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.725958 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c370799a55b474080d93cb980cf5f39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:17.729751 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896419bd407f43e1a0a361330ff9821b consensus state has no leader
I20250625 07:52:17.731637 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_87717ced5eb64d5fb6553780090af598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.732285 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87717ced5eb64d5fb6553780090af598 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.732369 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 012df7beedb1446ca5e533d6624afc10 with delete_type TABLET_DATA_DELETED (replaced by tablet aec412a15b964060a24631e7ebcf1ef7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.733608 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_071bd7e6cf2a4822b9a762cae07ceb4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.734127 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_071bd7e6cf2a4822b9a762cae07ceb4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.734149 1160 raft_consensus.cc:2270] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.734784 1160 ts_tablet_manager.cc:1905] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.735271 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_49c77ad274eb4800b6cafdeafc645b7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.735702 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49c77ad274eb4800b6cafdeafc645b7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.729301 1163 ts_tablet_manager.cc:1428] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
W20250625 07:52:17.737041 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c340466a42e47dead226f74c8176f49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.737534 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.737882 1166 tablet_replica.cc:331] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.738158 1166 raft_consensus.cc:2241] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.738400 1166 raft_consensus.cc:2270] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.742427 1163 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:17.743198 1166 ts_tablet_manager.cc:1905] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.744233 1163 tablet_bootstrap.cc:654] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:17.745378 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76fe692067954856908f936ced332c18 consensus state has no leader
I20250625 07:52:17.748508 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:17.749171 593 catalog_manager.cc:6652] ReplaceTablet: tablet 743123ffcf8240dc82bea7178559b7ea deleted and replaced by tablet 90f2bef7baa54f1a96b3b59921d59457
I20250625 07:52:17.749393 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.749738 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.750319 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.753255 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1786d469d14e46baa8b15791a16c0456" } new_table_name: "default.Table_3c49918d7bbe4d19978caf9281364386" modify_external_catalogs: true
I20250625 07:52:17.755103 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4bd7f1d95c4a4f62815742322b59570c" } new_table_name: "default.Table_40d2beb409354e9ab21f0069193dbe48" modify_external_catalogs: true
I20250625 07:52:17.758920 1162 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.759512 1166 ts_tablet_manager.cc:1918] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.759629 1166 log.cc:1199] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:17.760093 1166 ts_tablet_manager.cc:1939] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.767834 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (table default.Table_3c1cbfecab414abaa74c66f1773f817e [id=5174ca174ab24d7695eb1fddc9414c74]) successfully deleted
W20250625 07:52:17.768878 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 743123ffcf8240dc82bea7178559b7ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 743123ffcf8240dc82bea7178559b7ea already in progress: creating tablet
I20250625 07:52:17.770658 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c3505358432d40c19b748a55dad2af69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.771301 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3505358432d40c19b748a55dad2af69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.773938 1161 ts_tablet_manager.cc:1918] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.774111 1161 log.cc:1199] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1222f2c1027b42c6ad9627359a93d678
I20250625 07:52:17.775547 1158 tablet_bootstrap.cc:492] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.775872 1158 ts_tablet_manager.cc:1397] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.184s user 0.034s sys 0.038s
I20250625 07:52:17.777112 1158 raft_consensus.cc:357] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.777539 1158 raft_consensus.cc:383] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.777829 1158 raft_consensus.cc:738] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.778151 1158 consensus_queue.cc:260] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.779206 1158 ts_tablet_manager.cc:1428] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:17.779563 1158 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.780346 1308 raft_consensus.cc:491] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.780524 1308 raft_consensus.cc:513] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.781208 1158 tablet_bootstrap.cc:654] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.781205 1308 leader_election.cc:290] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:17.782287 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c0edbf6cbdb49a6b3d3fff7f347c8ce" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:17.783146 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c0edbf6cbdb49a6b3d3fff7f347c8ce" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:17.785852 850 leader_election.cc:343] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:17.786365 851 leader_election.cc:343] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.786482 851 leader_election.cc:304] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.787353 1308 raft_consensus.cc:2747] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.787762 1161 ts_tablet_manager.cc:1939] T 1222f2c1027b42c6ad9627359a93d678 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.788453 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 309) took 1512 ms. Trace:
I20250625 07:52:17.788655 1161 rpcz_store.cc:276] 0625 07:52:16.277188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.312444 (+ 35256us) service_pool.cc:225] Handling call
0625 07:52:17.723506 (+1411062us) ts_tablet_manager.cc:1145] Deleting tablet 1222f2c1027b42c6ad9627359a93d678
0625 07:52:17.723515 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.760318 (+ 36803us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.773806 (+ 13488us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.788375 (+ 14569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1393099}
I20250625 07:52:17.789927 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 324) took 1347 ms. Trace:
I20250625 07:52:17.790485 1162 tablet_bootstrap.cc:654] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.792140 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_43f3b349ae154f228f90a77e3c33c46d" } modify_external_catalogs: true
I20250625 07:52:17.792325 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_43f3b349ae154f228f90a77e3c33c46d" } modify_external_catalogs: true
I20250625 07:52:17.793831 597 master_service.cc:948] ReplaceTablet: received request to replace tablet e76df9405d91488dad0cfa7f8755bb1c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.794453 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aa09e516c85c4bc0abcb6bead66d3be1" } modify_external_catalogs: true
I20250625 07:52:17.794495 1163 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:17.794767 1163 ts_tablet_manager.cc:1397] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250625 07:52:17.795661 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aa09e516c85c4bc0abcb6bead66d3be1" } modify_external_catalogs: true
I20250625 07:52:17.795900 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_97384ebbc1584955af0a6bc425084522" } modify_external_catalogs: true
I20250625 07:52:17.796203 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_97384ebbc1584955af0a6bc425084522" } modify_external_catalogs: true
I20250625 07:52:17.796821 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c121aa9cbde14c6bb4a4bcb574484208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.797226 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 716e0c4dab584788bbb7631b4ca934bf from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:17.797370 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c121aa9cbde14c6bb4a4bcb574484208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.797822 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b7139335c7124ab79bc2d0a251984199 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.790256 1161 rpcz_store.cc:276] 0625 07:52:16.442700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.442749 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:17.789566 (+1346817us) ts_tablet_manager.cc:1145] Deleting tablet e289ea5553ac4ff983871ece4fc72ee5
0625 07:52:17.789577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.789784 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300027}
I20250625 07:52:17.799501 1158 tablet_bootstrap.cc:492] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.799666 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 327) took 1309 ms. Trace:
I20250625 07:52:17.799681 1158 ts_tablet_manager.cc:1397] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20250625 07:52:17.799777 1161 rpcz_store.cc:276] 0625 07:52:16.490015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.490069 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:17.799370 (+1309301us) ts_tablet_manager.cc:1145] Deleting tablet 6239713a586d43c28c101ba2447d579d
0625 07:52:17.799406 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.799583 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308388}
I20250625 07:52:17.800176 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 326) took 1317 ms. Trace:
I20250625 07:52:17.800277 1161 rpcz_store.cc:276] 0625 07:52:16.482221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.482898 (+ 677us) service_pool.cc:225] Handling call
0625 07:52:17.800015 (+1317117us) ts_tablet_manager.cc:1145] Deleting tablet 4fdb6a689ce6440ab7037484e2b9cdc5
0625 07:52:17.800022 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.800123 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1307500}
I20250625 07:52:17.800172 1163 raft_consensus.cc:357] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.800513 1158 raft_consensus.cc:357] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.800642 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 325) took 1340 ms. Trace:
I20250625 07:52:17.800724 1158 raft_consensus.cc:383] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.800751 1161 rpcz_store.cc:276] 0625 07:52:16.459751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.459828 (+ 77us) service_pool.cc:225] Handling call
0625 07:52:17.800504 (+1340676us) ts_tablet_manager.cc:1145] Deleting tablet cbdbd9230210455ba615d51f1a98267b
0625 07:52:17.800514 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.800590 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306909}
I20250625 07:52:17.800846 1158 raft_consensus.cc:738] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.800865 1163 raft_consensus.cc:383] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.801136 1163 raft_consensus.cc:738] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.801101 1158 consensus_queue.cc:260] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.801662 1158 ts_tablet_manager.cc:1428] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:17.801791 1163 consensus_queue.cc:260] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:17.802826 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e289ea5553ac4ff983871ece4fc72ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e289ea5553ac4ff983871ece4fc72ee5 already in progress: creating tablet
I20250625 07:52:17.803001 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 328) took 1310 ms. Trace:
W20250625 07:52:17.803081 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6239713a586d43c28c101ba2447d579d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6239713a586d43c28c101ba2447d579d already in progress: creating tablet
I20250625 07:52:17.803102 1161 rpcz_store.cc:276] 0625 07:52:16.492991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.493140 (+ 149us) service_pool.cc:225] Handling call
0625 07:52:17.800986 (+1307846us) ts_tablet_manager.cc:1145] Deleting tablet 6e5a207d515449e298936d8bb532437a
0625 07:52:17.802809 (+ 1823us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.802937 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1301435}
W20250625 07:52:17.803299 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fdb6a689ce6440ab7037484e2b9cdc5 already in progress: creating tablet
W20250625 07:52:17.803499 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet cbdbd9230210455ba615d51f1a98267b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbdbd9230210455ba615d51f1a98267b already in progress: creating tablet
W20250625 07:52:17.804279 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6e5a207d515449e298936d8bb532437a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e5a207d515449e298936d8bb532437a already in progress: creating tablet
I20250625 07:52:17.804774 1158 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.807381 1158 tablet_bootstrap.cc:654] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.808508 1161 tablet_replica.cc:331] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.808698 1161 raft_consensus.cc:2241] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.808908 1161 raft_consensus.cc:2270] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.809417 1161 ts_tablet_manager.cc:1905] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.810173 1373 raft_consensus.cc:491] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.810349 1373 raft_consensus.cc:513] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.811024 1373 leader_election.cc:290] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:17.811022 1163 ts_tablet_manager.cc:1428] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
W20250625 07:52:17.812021 986 leader_election.cc:343] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3338d1c708b4421c8501740da89afec5
I20250625 07:52:17.814965 1163 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:17.815840 985 leader_election.cc:343] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3338d1c708b4421c8501740da89afec5
I20250625 07:52:17.815979 985 leader_election.cc:304] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:17.816527 1373 raft_consensus.cc:2747] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.818964 1308 raft_consensus.cc:491] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.819124 1308 raft_consensus.cc:513] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:17.819759 1308 leader_election.cc:290] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:17.820355 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:17.819979 1160 ts_tablet_manager.cc:1918] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:17.821226 851 leader_election.cc:343] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.821326 1160 log.cc:1199] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4384bc544af149338514d40adf683f34
I20250625 07:52:17.822220 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:17.822647 1158 tablet_bootstrap.cc:492] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:17.822806 1158 ts_tablet_manager.cc:1397] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
W20250625 07:52:17.822939 850 leader_election.cc:343] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.823083 850 leader_election.cc:304] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.823565 1308 raft_consensus.cc:2747] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.823956 1158 raft_consensus.cc:357] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.825970 1333 raft_consensus.cc:491] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.826130 1333 raft_consensus.cc:513] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:17.826627 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cffde34fbba34e44b67985d3480666c3 consensus state has no leader
I20250625 07:52:17.826839 1333 leader_election.cc:290] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:17.828436 589 master_service.cc:948] ReplaceTablet: received request to replace tablet b7139335c7124ab79bc2d0a251984199 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.828596 1158 raft_consensus.cc:383] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.828737 1158 raft_consensus.cc:738] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.829005 1158 consensus_queue.cc:260] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.829563 1158 ts_tablet_manager.cc:1428] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 07:52:17.829905 1158 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:17.830359 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 365ef91e7bac43e283602d3d036853ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.830651 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:17.831773 1161 ts_tablet_manager.cc:1918] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.831738 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_405b7023434c463bbce1552ceb099b75" } modify_external_catalogs: true
I20250625 07:52:17.831931 1161 log.cc:1199] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/572b063d66434e4796f6b86f8221e3c9
I20250625 07:52:17.831943 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_405b7023434c463bbce1552ceb099b75" } modify_external_catalogs: true
I20250625 07:52:17.832324 1161 ts_tablet_manager.cc:1939] T 572b063d66434e4796f6b86f8221e3c9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.832882 1158 tablet_bootstrap.cc:654] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:17.833006 713 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.833209 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:17.835176 1162 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.835490 1162 ts_tablet_manager.cc:1397] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.000s
I20250625 07:52:17.836894 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cb6a73fc11c44f4ead125864cc91710d" } modify_external_catalogs: true
I20250625 07:52:17.837095 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cb6a73fc11c44f4ead125864cc91710d" } modify_external_catalogs: true
I20250625 07:52:17.837069 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:17.837303 915 raft_consensus.cc:2466] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:17.837353 1162 raft_consensus.cc:357] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.837869 713 leader_election.cc:304] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.837950 1160 ts_tablet_manager.cc:1939] T 4384bc544af149338514d40adf683f34 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:17.838299 1333 raft_consensus.cc:2802] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:17.838462 1333 raft_consensus.cc:491] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:17.838572 1333 raft_consensus.cc:3058] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:17.839334 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 86ea9df068a94d9187f2f9620e371a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.841370 1333 raft_consensus.cc:513] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:17.842631 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 329) took 1348 ms. Trace:
I20250625 07:52:17.842779 1161 rpcz_store.cc:276] 0625 07:52:16.494314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.494400 (+ 86us) service_pool.cc:225] Handling call
0625 07:52:17.803330 (+1308930us) ts_tablet_manager.cc:1145] Deleting tablet 572b063d66434e4796f6b86f8221e3c9
0625 07:52:17.808440 (+ 5110us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.820385 (+ 11945us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.831619 (+ 11234us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.842577 (+ 10958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293273}
I20250625 07:52:17.843309 1161 tablet_replica.cc:331] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.843521 1161 raft_consensus.cc:2241] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.843776 1161 raft_consensus.cc:2270] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.844430 1161 ts_tablet_manager.cc:1905] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.844493 1163 tablet_bootstrap.cc:654] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.845501 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 572b063d66434e4796f6b86f8221e3c9 (table default.Table_601b836403094b4cb7106f74b03172e7 [id=35433098b4b74c788ef591a6d8b8c391]) successfully deleted
I20250625 07:52:17.847563 1333 leader_election.cc:290] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:17.847676 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:17.847683 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4bb58e159ce944079a5760e1acb82e57" } new_table_name: "default.Table_4c6dba196ca04a6ba70a9180b32eeec3" modify_external_catalogs: true
I20250625 07:52:17.847759 1162 raft_consensus.cc:383] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.847877 915 raft_consensus.cc:3058] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:17.847898 1162 raft_consensus.cc:738] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.848193 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e77df5e7b3447fdbdb1157e74cf8b6d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:17.849009 1162 consensus_queue.cc:260] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:17.850409 713 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.850677 915 raft_consensus.cc:2466] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:17.854007 713 leader_election.cc:304] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.854972 1333 raft_consensus.cc:2802] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 07:52:17.856319 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet faf049714a7e4d7eb947c29f272bbb7c consensus state has no leader
I20250625 07:52:17.857303 1161 ts_tablet_manager.cc:1918] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:17.857447 1161 log.cc:1199] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2d5b724ca63c436ba1ee776b8df3cec9
I20250625 07:52:17.856876 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7e6dadd5349f42d08e3e43a15ca0c845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.857838 1161 ts_tablet_manager.cc:1939] T 2d5b724ca63c436ba1ee776b8df3cec9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:17.858230 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e6dadd5349f42d08e3e43a15ca0c845 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.859232 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 330) took 1357 ms. Trace:
I20250625 07:52:17.859347 1161 rpcz_store.cc:276] 0625 07:52:16.501751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.501835 (+ 84us) service_pool.cc:225] Handling call
0625 07:52:17.843090 (+1341255us) ts_tablet_manager.cc:1145] Deleting tablet 2d5b724ca63c436ba1ee776b8df3cec9
0625 07:52:17.843239 (+ 149us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.851152 (+ 7913us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.857182 (+ 6030us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.859182 (+ 2000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320245}
I20250625 07:52:17.859884 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2d5b724ca63c436ba1ee776b8df3cec9 (table default.Table_a940f52db1ba4b5caa2137bf0827a978 [id=6417af82ee934d5390c2a8887e9fa17f]) successfully deleted
I20250625 07:52:17.861382 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (table default.Table_ac06645d7b71464ab4e4e92402c14a13 [id=9511cff29b2441689b47351734929304]). Expected version 1 got 0
I20250625 07:52:17.861527 1162 ts_tablet_manager.cc:1428] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250625 07:52:17.864439 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 335) took 1326 ms. Trace:
I20250625 07:52:17.864544 1161 rpcz_store.cc:276] 0625 07:52:16.537932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.538533 (+ 601us) service_pool.cc:225] Handling call
0625 07:52:17.864255 (+1325722us) ts_tablet_manager.cc:1145] Deleting tablet 32bd7cdef96946e88683e457c2f26c9b
0625 07:52:17.864266 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.864381 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322452}
W20250625 07:52:17.865070 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 32bd7cdef96946e88683e457c2f26c9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32bd7cdef96946e88683e457c2f26c9b already in progress: creating tablet
I20250625 07:52:17.865159 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:17.865187 1162 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.865542 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 339) took 1132 ms. Trace:
I20250625 07:52:17.865644 1161 rpcz_store.cc:276] 0625 07:52:16.732871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.747746 (+ 14875us) service_pool.cc:225] Handling call
0625 07:52:17.865387 (+1117641us) ts_tablet_manager.cc:1145] Deleting tablet 6b71aadf074a4a39943b2ff930909f93
0625 07:52:17.865399 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.865495 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117374}
I20250625 07:52:17.866117 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 340) took 1117 ms. Trace:
I20250625 07:52:17.866117 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.866197 1161 rpcz_store.cc:276] 0625 07:52:16.748235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.748287 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:17.865964 (+1117677us) ts_tablet_manager.cc:1145] Deleting tablet fc6c3be49be14ecbb3c26ef107ede015
0625 07:52:17.865986 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.866068 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117460}
W20250625 07:52:17.866969 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because the tablet was not found. No further retry: Not found: Tablet not found: 23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:17.866648 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 341) took 1118 ms. Trace:
I20250625 07:52:17.867144 1161 rpcz_store.cc:276] 0625 07:52:16.748633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.748713 (+ 80us) service_pool.cc:225] Handling call
0625 07:52:17.866520 (+1117807us) ts_tablet_manager.cc:1145] Deleting tablet cd728ffc2a274deea3fad533d1eaa6da
0625 07:52:17.866528 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.866595 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117574}
W20250625 07:52:17.867525 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6b71aadf074a4a39943b2ff930909f93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b71aadf074a4a39943b2ff930909f93 already in progress: creating tablet
I20250625 07:52:17.867682 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:17.867770 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fc6c3be49be14ecbb3c26ef107ede015 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc6c3be49be14ecbb3c26ef107ede015 already in progress: creating tablet
W20250625 07:52:17.868005 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet cd728ffc2a274deea3fad533d1eaa6da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd728ffc2a274deea3fad533d1eaa6da already in progress: creating tablet
I20250625 07:52:17.868048 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 342) took 1115 ms. Trace:
I20250625 07:52:17.868129 1161 rpcz_store.cc:276] 0625 07:52:16.752882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.752931 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:17.867898 (+1114967us) ts_tablet_manager.cc:1145] Deleting tablet c263b671be384cd097041c60306034de
0625 07:52:17.867906 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.867997 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1114697}
W20250625 07:52:17.868530 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c263b671be384cd097041c60306034de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c263b671be384cd097041c60306034de already in progress: creating tablet
I20250625 07:52:17.869033 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 343) took 1113 ms. Trace:
I20250625 07:52:17.869153 1161 rpcz_store.cc:276] 0625 07:52:16.755509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.755564 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:17.868803 (+1113239us) ts_tablet_manager.cc:1145] Deleting tablet 2f63fdf2c8464c8fb58f412263d4e5c5
0625 07:52:17.868815 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.868962 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113014}
I20250625 07:52:17.869319 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:17.869632 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f63fdf2c8464c8fb58f412263d4e5c5 already in progress: creating tablet
I20250625 07:52:17.870762 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (table default.Table_ac06645d7b71464ab4e4e92402c14a13 [id=9511cff29b2441689b47351734929304]). Expected version 1 got 0
I20250625 07:52:17.871981 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:17.872627 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.873145 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2bcfe7c3c54741fbb0aca69650fc8eee
W20250625 07:52:17.873289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29eaa8d93e5b4d1d8ed147b60c7da859 already in progress: creating tablet
I20250625 07:52:17.873885 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.874456 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because the tablet was not found. No further retry: Not found: Tablet not found: 2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:17.875618 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.875803 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.875943 1161 tablet_replica.cc:331] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.876166 1161 raft_consensus.cc:2241] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.876374 1161 raft_consensus.cc:2270] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.876976 1161 ts_tablet_manager.cc:1905] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.877460 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.877732 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.877774 1162 tablet_bootstrap.cc:654] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:17.877936 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.883040 666 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 23af43b9ceeb4f2b812ee251ea8545ce
W20250625 07:52:17.883767 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1770f6d5d1d4899a66ec23ab4eadc35 consensus state has no leader
I20250625 07:52:17.884104 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.884227 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 716e0c4dab584788bbb7631b4ca934bf
I20250625 07:52:17.884367 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.884660 590 catalog_manager.cc:6652] ReplaceTablet: tablet 716e0c4dab584788bbb7631b4ca934bf deleted and replaced by tablet e181a3f1ae2c49349718d842c166b1e6
I20250625 07:52:17.885358 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.885629 666 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:17.886018 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.886291 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.886495 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.886746 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.886770 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.887112 1166 tablet_replica.cc:331] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.887308 1166 raft_consensus.cc:2241] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.887406 1308 raft_consensus.cc:491] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.887475 1166 raft_consensus.cc:2270] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.887555 1308 raft_consensus.cc:513] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.888021 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.888211 1308 leader_election.cc:290] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:17.888653 1166 ts_tablet_manager.cc:1905] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.888834 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f62b03db94d4ee3935866b5dd8bd53f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:17.888892 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:17.889322 851 leader_election.cc:343] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.889652 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.890272 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f62b03db94d4ee3935866b5dd8bd53f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:17.891991 850 leader_election.cc:343] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.892127 850 leader_election.cc:304] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.892529 1308 raft_consensus.cc:2747] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.894656 892 tablet_service.cc:1468] Processing CreateTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce (DEFAULT_TABLE table=default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.895087 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23af43b9ceeb4f2b812ee251ea8545ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.895327 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce (DEFAULT_TABLE table=default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.895697 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23af43b9ceeb4f2b812ee251ea8545ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.901589 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=94 overwritten=0 applied=93 ignored=0} inserts{seen=87 ignored=0} mutations{seen=95 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250625 07:52:17.901597 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e8d65c2ed5240fc923a7c7b59488e6a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:17.901638 1308 raft_consensus.cc:491] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:17.901784 1308 raft_consensus.cc:513] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.902447 1308 leader_election.cc:290] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:17.903056 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:17.903597 851 leader_election.cc:343] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:17.904207 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:17.904731 850 leader_election.cc:343] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:17.904886 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because the tablet was not found. No further retry: Not found: Tablet not found: 23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:17.904942 850 leader_election.cc:304] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:17.905565 1308 raft_consensus.cc:2747] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:17.906044 747 tablet_service.cc:1468] Processing CreateTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce (DEFAULT_TABLE table=default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.906474 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23af43b9ceeb4f2b812ee251ea8545ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.906489 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap complete.
I20250625 07:52:17.907153 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9418b81f010d4f369d3409e9364bc5fa
I20250625 07:52:17.908313 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:17.909024 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.913697 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.913861 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:17.914461 744 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.914682 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:17.915503 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4384bc544af149338514d40adf683f34 (table default.Table_ddae42a67be34593a367bf6f3d9e8cde [id=fd174ee9d81e42c19cab3c058584ea20]) successfully deleted
I20250625 07:52:17.916100 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.916966 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:17.917871 1161 ts_tablet_manager.cc:1918] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.918045 1161 log.cc:1199] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/d51a898202154881ab4ad41da1d519a3
I20250625 07:52:17.918143 1166 ts_tablet_manager.cc:1918] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.918282 1166 log.cc:1199] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:17.918517 1161 ts_tablet_manager.cc:1939] T d51a898202154881ab4ad41da1d519a3 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.918673 1166 ts_tablet_manager.cc:1939] T 8e8d65c2ed5240fc923a7c7b59488e6a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:17.919152 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 8e8d65c2ed5240fc923a7c7b59488e6a (table default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]) successfully deleted
I20250625 07:52:17.919497 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 349) took 1070 ms. Trace:
I20250625 07:52:17.919550 756 tablet_service.cc:1468] Processing CreateTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 (DEFAULT_TABLE table=default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.919656 1161 rpcz_store.cc:276] 0625 07:52:16.849234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.849365 (+ 131us) service_pool.cc:225] Handling call
0625 07:52:17.875862 (+1026497us) ts_tablet_manager.cc:1145] Deleting tablet d51a898202154881ab4ad41da1d519a3
0625 07:52:17.875872 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.880528 (+ 4656us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.917680 (+ 37152us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.919442 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026181}
I20250625 07:52:17.919955 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a9f4cf6838411fab06fa6d35ae9492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.920138 1161 tablet_replica.cc:331] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.920413 1161 raft_consensus.cc:2241] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.920384 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 (DEFAULT_TABLE table=default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.920653 1161 raft_consensus.cc:2270] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.920840 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a9f4cf6838411fab06fa6d35ae9492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.921375 1161 ts_tablet_manager.cc:1905] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.925097 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet d51a898202154881ab4ad41da1d519a3 (table default.Table_9e0159dbcce04493a7cc2849bde83764 [id=21758be8aff9485b9313da6b33405f8e]) successfully deleted
W20250625 07:52:17.929376 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 because the tablet was not found. No further retry: Not found: Tablet not found: 4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:17.916325 1440 raft_consensus.cc:357] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:17.930536 892 tablet_service.cc:1468] Processing CreateTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 (DEFAULT_TABLE table=default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.930553 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee (DEFAULT_TABLE table=default.Table_aa09e516c85c4bc0abcb6bead66d3be1 [id=0d08c43a2d7a432391933600b06ce3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.931046 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a9f4cf6838411fab06fa6d35ae9492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.931411 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcfe7c3c54741fbb0aca69650fc8eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.931775 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee (DEFAULT_TABLE table=default.Table_aa09e516c85c4bc0abcb6bead66d3be1 [id=0d08c43a2d7a432391933600b06ce3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.934114 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcfe7c3c54741fbb0aca69650fc8eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.934614 1160 tablet_replica.cc:331] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:17.934738 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because the tablet was not found. No further retry: Not found: Tablet not found: 2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:17.934999 1160 raft_consensus.cc:2241] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.935252 1160 raft_consensus.cc:2270] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.935962 1160 ts_tablet_manager.cc:1905] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:17.936939 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 716e0c4dab584788bbb7631b4ca934bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 716e0c4dab584788bbb7631b4ca934bf already in progress: creating tablet
I20250625 07:52:17.936969 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a414bf3d9fc24f58ac403f02e4994a8f" } new_table_name: "default.Table_929703cfa29e467c9090ff84a8b2cec1" modify_external_catalogs: true
I20250625 07:52:17.938163 1166 tablet_replica.cc:331] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:17.939714 1166 raft_consensus.cc:2241] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.940415 1440 raft_consensus.cc:738] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3faeffaa4422ea73b98ee74768a4a, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.942260 1440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.94, Last appended by leader: 94, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
W20250625 07:52:17.943920 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a630f8804bbf41de86b08cde419643ff consensus state has no leader
I20250625 07:52:17.943739 1166 raft_consensus.cc:2270] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:17.944980 1161 ts_tablet_manager.cc:1918] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:17.946770 1161 log.cc:1199] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/239e99c6b91d41d68a213a4c8d79470f
I20250625 07:52:17.948292 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eea0319c78844065b1dc101bb7ff333c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.948349 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c49918d7bbe4d19978caf9281364386" } new_table_name: "default.Table_6aeac87c40794fa28a9541a027c60bcd" modify_external_catalogs: true
W20250625 07:52:17.948993 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eea0319c78844065b1dc101bb7ff333c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.948886 1166 ts_tablet_manager.cc:1905] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:17.950634 1161 ts_tablet_manager.cc:1939] T 239e99c6b91d41d68a213a4c8d79470f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:17.952531 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d057d5343c334e9e92d5d9d977f90e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.953265 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d057d5343c334e9e92d5d9d977f90e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:17.955610 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1770f6d5d1d4899a66ec23ab4eadc35 consensus state has no leader
I20250625 07:52:17.955667 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 350) took 1076 ms. Trace:
I20250625 07:52:17.957036 1162 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:17.959043 1162 ts_tablet_manager.cc:1397] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.004s
I20250625 07:52:17.957130 1161 rpcz_store.cc:276] 0625 07:52:16.878426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.878820 (+ 394us) service_pool.cc:225] Handling call
0625 07:52:17.920028 (+1041208us) ts_tablet_manager.cc:1145] Deleting tablet 239e99c6b91d41d68a213a4c8d79470f
0625 07:52:17.920041 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.926580 (+ 6539us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.943875 (+ 17295us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:17.955069 (+ 11194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040982}
I20250625 07:52:17.959873 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee (DEFAULT_TABLE table=default.Table_aa09e516c85c4bc0abcb6bead66d3be1 [id=0d08c43a2d7a432391933600b06ce3fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.960347 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcfe7c3c54741fbb0aca69650fc8eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.962342 1161 tablet_replica.cc:331] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:17.966358 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:17.966698 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:17.970027 1440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:52:17.970651 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 239e99c6b91d41d68a213a4c8d79470f (table default.Table_e0125dcbb680481899e4adad86d7c2b5 [id=0d85f054a1f04f63ae79d8514c4c978b]) successfully deleted
I20250625 07:52:17.973060 747 tablet_service.cc:1468] Processing CreateTablet for tablet ffd98b2877b944f0bd484efa2a38a00a (DEFAULT_TABLE table=default.Table_701b07f0889e4b3a8765f597fa055673 [id=74ad120db7074f45acf606261432f7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.973574 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd98b2877b944f0bd484efa2a38a00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.974712 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3823279d7e06435484f1b82836ebcf39 (DEFAULT_TABLE table=default.Table_5ad0a090d681485683aa607f6f29b2b1 [id=9197109c61c648aeb670a20ef196bb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.975113 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3823279d7e06435484f1b82836ebcf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.982571 887 tablet_service.cc:1468] Processing CreateTablet for tablet ffd98b2877b944f0bd484efa2a38a00a (DEFAULT_TABLE table=default.Table_701b07f0889e4b3a8765f597fa055673 [id=74ad120db7074f45acf606261432f7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.983167 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd98b2877b944f0bd484efa2a38a00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.986687 1162 raft_consensus.cc:357] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.988034 1012 tablet_service.cc:1468] Processing CreateTablet for tablet ffd98b2877b944f0bd484efa2a38a00a (DEFAULT_TABLE table=default.Table_701b07f0889e4b3a8765f597fa055673 [id=74ad120db7074f45acf606261432f7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.988724 1162 raft_consensus.cc:383] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:17.989116 1162 raft_consensus.cc:738] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:17.989681 1162 consensus_queue.cc:260] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:17.990031 1161 raft_consensus.cc:2241] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:17.990156 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffd98b2877b944f0bd484efa2a38a00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.990973 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3823279d7e06435484f1b82836ebcf39 (DEFAULT_TABLE table=default.Table_5ad0a090d681485683aa607f6f29b2b1 [id=9197109c61c648aeb670a20ef196bb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.991077 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3823279d7e06435484f1b82836ebcf39 (DEFAULT_TABLE table=default.Table_5ad0a090d681485683aa607f6f29b2b1 [id=9197109c61c648aeb670a20ef196bb30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:17.991406 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3823279d7e06435484f1b82836ebcf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:17.992448 1162 ts_tablet_manager.cc:1428] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250625 07:52:17.993549 1162 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:17.998071 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5a458f32a2864c59a2c926d7760d2f13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:17.998127 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f801c447326646d1ab93ac87fff7843a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.998750 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f801c447326646d1ab93ac87fff7843a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:17.999050 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f00b99b9279946bab7c178245d31c414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:17.999775 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f00b99b9279946bab7c178245d31c414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:17.998754 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a458f32a2864c59a2c926d7760d2f13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.001387 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5e910ce70a934be2be117a115fea3380"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.002048 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e910ce70a934be2be117a115fea3380 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.003742 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e1eb60c1e8c146a596391600eaffc9d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.004172 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1eb60c1e8c146a596391600eaffc9d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.008060 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3823279d7e06435484f1b82836ebcf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.009372 747 tablet_service.cc:1468] Processing CreateTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.009821 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d2bb8b0a8f45079083f2b42bc25011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.014760 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.015257 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d2bb8b0a8f45079083f2b42bc25011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.017236 744 tablet_service.cc:1468] Processing CreateTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.017642 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f47caf3bae41b79b0aa1e320f15da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.021857 1161 raft_consensus.cc:2270] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.022558 1161 ts_tablet_manager.cc:1905] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.028206 1333 raft_consensus.cc:695] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:18.028206 889 tablet_service.cc:1468] Processing CreateTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.028745 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d2bb8b0a8f45079083f2b42bc25011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.029003 1333 consensus_queue.cc:237] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.030138 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4c0fff3cc3fe4f31a88c71012c45b906" } new_table_name: "default.Table_6288489a16da45cf8a64efae2aa3f532" modify_external_catalogs: true
I20250625 07:52:18.036657 1162 tablet_bootstrap.cc:654] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.040153 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e8d65c2ed5240fc923a7c7b59488e6a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.041563 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_458ec3bda21d4689bdd0cfa9ac3e6dc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.042395 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_458ec3bda21d4689bdd0cfa9ac3e6dc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.044407 590 catalog_manager.cc:5582] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:18.046483 1166 ts_tablet_manager.cc:1918] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.046672 1166 log.cc:1199] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9e800d4982fa45f4b931985c8b52719d
I20250625 07:52:18.050174 1166 ts_tablet_manager.cc:1939] T 9e800d4982fa45f4b931985c8b52719d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.050323 1163 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:18.050418 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e109c9dca7444681ac517c7fc3e14dfa consensus state has no leader
I20250625 07:52:18.051427 1163 ts_tablet_manager.cc:1397] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.238s user 0.004s sys 0.001s
I20250625 07:52:18.052482 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_06060aabdbd54758b16137634d989c0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.052754 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9e800d4982fa45f4b931985c8b52719d (table default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]) successfully deleted
W20250625 07:52:18.053267 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06060aabdbd54758b16137634d989c0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.054975 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b2bb060cdd9a48688967989d7491cfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bb060cdd9a48688967989d7491cfc1 already in progress: creating tablet
I20250625 07:52:18.056476 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 (DEFAULT_TABLE table=default.Table_baabb7e0fd1e496bbe725e5e00574146 [id=c55921e26548445d9872a9e958f36056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.056929 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70094e6c7e644f62bf5ef04dcfb4c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.058202 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.058890 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0bee6f682a14fa59118497e9a616ebf" } new_table_name: "default.Table_b3b5ccfb28464dcda7e964281f02f69a" modify_external_catalogs: true
I20250625 07:52:18.055239 884 tablet_service.cc:1468] Processing CreateTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.060500 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f47caf3bae41b79b0aa1e320f15da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.064111 756 tablet_service.cc:1468] Processing CreateTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 (DEFAULT_TABLE table=default.Table_baabb7e0fd1e496bbe725e5e00574146 [id=c55921e26548445d9872a9e958f36056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.064558 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70094e6c7e644f62bf5ef04dcfb4c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.065709 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f47caf3bae41b79b0aa1e320f15da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.069619 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c" } modify_external_catalogs: true
I20250625 07:52:18.069857 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c" } modify_external_catalogs: true
W20250625 07:52:18.073319 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b7daf759e324c1db853fdbaab76c721 consensus state has no leader
I20250625 07:52:18.075140 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_41d1f73c2d534e5496df588509888b0f" } modify_external_catalogs: true
I20250625 07:52:18.075522 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_41d1f73c2d534e5496df588509888b0f" } modify_external_catalogs: true
I20250625 07:52:18.078037 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f89a55bc2694564b94bb87f42761d2a" } modify_external_catalogs: true
I20250625 07:52:18.078295 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f89a55bc2694564b94bb87f42761d2a" } modify_external_catalogs: true
W20250625 07:52:18.079712 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9418b81f010d4f369d3409e9364bc5fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9418b81f010d4f369d3409e9364bc5fa already in progress: creating tablet
I20250625 07:52:18.080248 1161 ts_tablet_manager.cc:1918] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.080408 1161 log.cc:1199] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2b020ff9f04b4d1796af4ff585cc04aa
I20250625 07:52:18.080482 1158 tablet_bootstrap.cc:492] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:18.080683 1158 ts_tablet_manager.cc:1397] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.251s user 0.033s sys 0.047s
I20250625 07:52:18.081182 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9f7e1cd61b2d4a87b378410741a2a5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.081744 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f7e1cd61b2d4a87b378410741a2a5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.083343 1163 raft_consensus.cc:357] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.083812 1163 raft_consensus.cc:383] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.083979 1163 raft_consensus.cc:738] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.084450 1163 consensus_queue.cc:260] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.080904 1161 ts_tablet_manager.cc:1939] T 2b020ff9f04b4d1796af4ff585cc04aa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:18.090318 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 351) took 1185 ms. Trace:
I20250625 07:52:18.090485 1158 raft_consensus.cc:357] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.090516 1161 rpcz_store.cc:276] 0625 07:52:16.904878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:16.909377 (+ 4499us) service_pool.cc:225] Handling call
0625 07:52:17.961152 (+1051775us) ts_tablet_manager.cc:1145] Deleting tablet 2b020ff9f04b4d1796af4ff585cc04aa
0625 07:52:17.961166 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.036990 (+ 75824us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.054164 (+ 17174us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.090264 (+ 36100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032673}
I20250625 07:52:18.090690 1158 raft_consensus.cc:383] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.090839 1158 raft_consensus.cc:738] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.091115 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 359) took 1044 ms. Trace:
I20250625 07:52:18.091240 1161 rpcz_store.cc:276] 0625 07:52:17.046942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.046998 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:18.090883 (+1043885us) ts_tablet_manager.cc:1145] Deleting tablet 06e4bf78218f4b82b4f16f59b1704256
0625 07:52:18.090893 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.091065 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043663}
I20250625 07:52:18.091658 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 360) took 1039 ms. Trace:
I20250625 07:52:18.091768 1161 rpcz_store.cc:276] 0625 07:52:17.052380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.052436 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:18.091515 (+1039079us) ts_tablet_manager.cc:1145] Deleting tablet 29f2997d1d714100950383e6af04f315
0625 07:52:18.091523 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.091626 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038842}
I20250625 07:52:18.092110 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 363) took 1032 ms. Trace:
I20250625 07:52:18.092233 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2b020ff9f04b4d1796af4ff585cc04aa (table default.Table_20ee9c20a0714e32b1a10f14a48df181 [id=c0b762f0272b44ba935f0925ce72fc19]) successfully deleted
I20250625 07:52:18.092224 1161 rpcz_store.cc:276] 0625 07:52:17.059421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.059470 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:18.091986 (+1032516us) ts_tablet_manager.cc:1145] Deleting tablet 49697a10cdac4c8ebb69f10f4c3437ec
0625 07:52:18.091995 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.092074 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032293}
W20250625 07:52:18.092502 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 06e4bf78218f4b82b4f16f59b1704256 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e4bf78218f4b82b4f16f59b1704256 already in progress: creating tablet
I20250625 07:52:18.092964 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 364) took 1032 ms. Trace:
W20250625 07:52:18.093103 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 29f2997d1d714100950383e6af04f315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29f2997d1d714100950383e6af04f315 already in progress: creating tablet
W20250625 07:52:18.093366 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 49697a10cdac4c8ebb69f10f4c3437ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49697a10cdac4c8ebb69f10f4c3437ec already in progress: creating tablet
I20250625 07:52:18.094420 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b5f917cfd2384a72920847f8b89358b9
I20250625 07:52:18.093125 1161 rpcz_store.cc:276] 0625 07:52:17.060286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.060523 (+ 237us) service_pool.cc:225] Handling call
0625 07:52:18.092430 (+1031907us) ts_tablet_manager.cc:1145] Deleting tablet 4c0bbd21d4aa41b2b92473bffe7dc05a
0625 07:52:18.092439 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.092929 (+ 490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031669}
I20250625 07:52:18.095852 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 365) took 1014 ms. Trace:
I20250625 07:52:18.095990 1161 rpcz_store.cc:276] 0625 07:52:17.081390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.081443 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:18.095685 (+1014242us) ts_tablet_manager.cc:1145] Deleting tablet a3670cadfa2841949705c65c9b6c0927
0625 07:52:18.095695 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.095804 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013946}
I20250625 07:52:18.097010 756 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:18.097725 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b5f917cfd2384a72920847f8b89358b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f917cfd2384a72920847f8b89358b9 already in progress: creating tablet
I20250625 07:52:18.098168 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9cc1842f257d409ba0a56873f582f023
I20250625 07:52:18.098896 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:18.099634 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c0bbd21d4aa41b2b92473bffe7dc05a already in progress: creating tablet
W20250625 07:52:18.099859 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a3670cadfa2841949705c65c9b6c0927 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3670cadfa2841949705c65c9b6c0927 already in progress: creating tablet
W20250625 07:52:18.100551 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9cc1842f257d409ba0a56873f582f023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cc1842f257d409ba0a56873f582f023 already in progress: creating tablet
I20250625 07:52:18.100823 895 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.099908 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.101052 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.100004 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.101230 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.098384 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 366) took 1010 ms. Trace:
I20250625 07:52:18.102449 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b71aadf074a4a39943b2ff930909f93 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc1842f257d409ba0a56873f582f023) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.102509 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d6ac142574f64d839444fe2bf4673aed" } new_table_name: "default.Table_3770979326894b6890fc810005775fc1" modify_external_catalogs: true
I20250625 07:52:18.102851 1166 tablet_replica.cc:331] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.103050 1166 raft_consensus.cc:2241] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.103220 1166 raft_consensus.cc:2270] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.101451 1161 rpcz_store.cc:276] 0625 07:52:17.088299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.088365 (+ 66us) service_pool.cc:225] Handling call
0625 07:52:18.098164 (+1009799us) ts_tablet_manager.cc:1145] Deleting tablet bbe5c37fc9bd49c685de8f469a7af0c6
0625 07:52:18.098175 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.098341 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009553}
W20250625 07:52:18.104064 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bbe5c37fc9bd49c685de8f469a7af0c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe5c37fc9bd49c685de8f469a7af0c6 already in progress: creating tablet
I20250625 07:52:18.104095 1163 ts_tablet_manager.cc:1428] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250625 07:52:18.104432 1161 tablet_replica.cc:331] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:18.104683 1161 raft_consensus.cc:2241] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.104950 1161 raft_consensus.cc:2270] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.105096 1166 ts_tablet_manager.cc:1905] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.105805 1161 ts_tablet_manager.cc:1905] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.111634 1166 ts_tablet_manager.cc:1918] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.111896 1166 log.cc:1199] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6b71aadf074a4a39943b2ff930909f93
I20250625 07:52:18.112347 1166 ts_tablet_manager.cc:1939] T 6b71aadf074a4a39943b2ff930909f93 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.112731 1161 ts_tablet_manager.cc:1918] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.115777 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3505358432d40c19b748a55dad2af69" } modify_external_catalogs: true
I20250625 07:52:18.116150 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3505358432d40c19b748a55dad2af69" } modify_external_catalogs: true
I20250625 07:52:18.116762 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6b71aadf074a4a39943b2ff930909f93 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:18.116956 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0d58b443a7b2466488c13f5ec50e1a0f" } modify_external_catalogs: true
I20250625 07:52:18.117228 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0d58b443a7b2466488c13f5ec50e1a0f" } modify_external_catalogs: true
I20250625 07:52:18.118127 1161 log.cc:1199] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6b3488a9fb19452dac3241170dd2fee2
I20250625 07:52:18.118165 1163 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.119432 1162 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.119624 1162 ts_tablet_manager.cc:1397] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.000s
I20250625 07:52:18.120436 1163 tablet_bootstrap.cc:654] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.120419 1162 raft_consensus.cc:357] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.120615 1162 raft_consensus.cc:383] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.120712 1162 raft_consensus.cc:738] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.120975 1162 consensus_queue.cc:260] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.121454 1162 ts_tablet_manager.cc:1428] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:18.121824 1162 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.123690 1162 tablet_bootstrap.cc:654] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.123939 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1796a3305cb94ccb9a0596643a736a96 (DEFAULT_TABLE table=default.Table_75a5d24d45144619ab16307ddb8a5807 [id=5cb03ca1f790465aba5b692876e4f6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.125797 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:18.128417 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1796a3305cb94ccb9a0596643a736a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.130666 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1796a3305cb94ccb9a0596643a736a96 (DEFAULT_TABLE table=default.Table_75a5d24d45144619ab16307ddb8a5807 [id=5cb03ca1f790465aba5b692876e4f6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.131171 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1796a3305cb94ccb9a0596643a736a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.134117 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:18.135296 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:18.138262 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:18.139330 745 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.139842 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:18.140041 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a33b7cab19244ec08d3821b6cc4c072d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a33b7cab19244ec08d3821b6cc4c072d already in progress: creating tablet
I20250625 07:52:18.140453 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.141108 884 tablet_service.cc:1468] Processing CreateTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 (DEFAULT_TABLE table=default.Table_baabb7e0fd1e496bbe725e5e00574146 [id=c55921e26548445d9872a9e958f36056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.141624 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70094e6c7e644f62bf5ef04dcfb4c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.145905 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5be7c764e2274992a4c2809f83a3dcea" } new_table_name: "default.Table_70dce4df11fe47c8b1ea0003e745f82d" modify_external_catalogs: true
I20250625 07:52:18.150174 1162 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.150439 1162 ts_tablet_manager.cc:1397] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.005s
I20250625 07:52:18.151329 1162 raft_consensus.cc:357] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.151634 1162 raft_consensus.cc:383] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.151782 1162 raft_consensus.cc:738] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.152135 1162 consensus_queue.cc:260] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.153388 1162 ts_tablet_manager.cc:1428] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:18.118752 1161 ts_tablet_manager.cc:1939] T 6b3488a9fb19452dac3241170dd2fee2 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:18.154139 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 367) took 1053 ms. Trace:
I20250625 07:52:18.154304 1161 rpcz_store.cc:276] 0625 07:52:17.100889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.100957 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:18.104360 (+1003403us) ts_tablet_manager.cc:1145] Deleting tablet 6b3488a9fb19452dac3241170dd2fee2
0625 07:52:18.104370 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.109376 (+ 5006us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.112604 (+ 3228us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.153947 (+ 41343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003168}
I20250625 07:52:18.154827 1161 tablet_replica.cc:331] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:18.154992 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6b3488a9fb19452dac3241170dd2fee2 (table default.Table_e407dc6ea5d247b1a9c26729116da0e2 [id=02ec3b2efaed40e7b6354e1f15d78043]) successfully deleted
I20250625 07:52:18.155073 1161 raft_consensus.cc:2241] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.155294 1161 raft_consensus.cc:2270] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.155920 1161 ts_tablet_manager.cc:1905] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.158679 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_24edc711d8fd46d7b8d3e7d20be99016" } new_table_name: "default.Table_0c0f428f2b54479a8e0bd1f36ef2dede" modify_external_catalogs: true
I20250625 07:52:18.158881 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1796a3305cb94ccb9a0596643a736a96 (DEFAULT_TABLE table=default.Table_75a5d24d45144619ab16307ddb8a5807 [id=5cb03ca1f790465aba5b692876e4f6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.159463 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1796a3305cb94ccb9a0596643a736a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.160403 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e362ea24c1794efbb1c667870df19840"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.161072 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e362ea24c1794efbb1c667870df19840 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.164090 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 61e076870efc4744b1224ddcb0eef3fa (DEFAULT_TABLE table=default.Table_f6885e8b33124109bb5ab1f1128522de [id=9a2fe4250951446188a9e591f8f38b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.124554 1158 consensus_queue.cc:260] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.164983 1373 raft_consensus.cc:491] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.164781 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e076870efc4744b1224ddcb0eef3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.165162 1373 raft_consensus.cc:513] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.165217 744 tablet_service.cc:1468] Processing CreateTablet for tablet 61e076870efc4744b1224ddcb0eef3fa (DEFAULT_TABLE table=default.Table_f6885e8b33124109bb5ab1f1128522de [id=9a2fe4250951446188a9e591f8f38b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.166007 1373 leader_election.cc:290] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:18.166589 1160 ts_tablet_manager.cc:1918] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.166716 1160 log.cc:1199] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5f62b03db94d4ee3935866b5dd8bd53f
I20250625 07:52:18.167083 1160 ts_tablet_manager.cc:1939] T 5f62b03db94d4ee3935866b5dd8bd53f P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:18.169955 885 tablet_service.cc:1468] Processing CreateTablet for tablet 61e076870efc4744b1224ddcb0eef3fa (DEFAULT_TABLE table=default.Table_f6885e8b33124109bb5ab1f1128522de [id=9a2fe4250951446188a9e591f8f38b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.170192 1333 raft_consensus.cc:491] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.170357 1333 raft_consensus.cc:513] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.170971 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e076870efc4744b1224ddcb0eef3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.171100 1333 leader_election.cc:290] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.176995 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:18.177286 915 raft_consensus.cc:2466] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:18.178443 713 leader_election.cc:304] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:18.178828 1333 raft_consensus.cc:2802] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:18.178948 1333 raft_consensus.cc:491] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:18.179066 1333 raft_consensus.cc:3058] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.181882 1333 raft_consensus.cc:513] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.183979 1333 leader_election.cc:290] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.184314 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:18.184566 915 raft_consensus.cc:3058] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.186913 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aada5cc96edc434b8e666d538d4eb5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.187574 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aada5cc96edc434b8e666d538d4eb5e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.200999 596 master_service.cc:948] ReplaceTablet: received request to replace tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.201670 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e076870efc4744b1224ddcb0eef3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.202757 1162 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.204591 1162 tablet_bootstrap.cc:654] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.207863 890 tablet_service.cc:1468] Processing CreateTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a (DEFAULT_TABLE table=default.Table_7fc614d790ab4feaa0e3167c5de7cffd [id=93204c5397784d4fa49cd180005e4d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.207937 756 tablet_service.cc:1468] Processing CreateTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a (DEFAULT_TABLE table=default.Table_7fc614d790ab4feaa0e3167c5de7cffd [id=93204c5397784d4fa49cd180005e4d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.208319 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.209283 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.221756 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 23755f70822740e08ca6a5e5b5571458 with delete_type TABLET_DATA_DELETED (replaced by tablet 81976db0a2f94128a89842a12f067231) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.222060 1158 ts_tablet_manager.cc:1428] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.095s user 0.001s sys 0.001s
I20250625 07:52:18.222472 1158 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:18.224399 1158 tablet_bootstrap.cc:654] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.228334 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9f2673ddf3e34a969d58c4514580bdcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.229032 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f2673ddf3e34a969d58c4514580bdcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.230542 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c62e57a0d535414a93d82dca7dd477ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.231124 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9e77df5e7b3447fdbdb1157e74cf8b6d
W20250625 07:52:18.231227 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62e57a0d535414a93d82dca7dd477ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.232533 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c934a46a8f6c407caa7a84d6415d6901 consensus state has no leader
W20250625 07:52:18.232936 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 547b758c2e6c4558bb232f14b37aa81a consensus state has no leader
I20250625 07:52:18.233078 1162 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.233251 1162 ts_tablet_manager.cc:1397] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.002s
I20250625 07:52:18.233768 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_380bf0b7bfc248f7b340c14b230c4db7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.234145 1162 raft_consensus.cc:357] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.234377 1162 raft_consensus.cc:383] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:18.234367 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_380bf0b7bfc248f7b340c14b230c4db7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.234310 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f98ca9d11df0483887149bfbb92885be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.234452 1162 raft_consensus.cc:738] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.234755 1162 consensus_queue.cc:260] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.235276 1162 ts_tablet_manager.cc:1428] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:18.236076 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:18.237092 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f98ca9d11df0483887149bfbb92885be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.238320 1162 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.238826 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.238951 1308 raft_consensus.cc:491] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.239130 1166 tablet_replica.cc:331] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.239167 1308 raft_consensus.cc:513] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.239326 1166 raft_consensus.cc:2241] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:18.239811 1166 pending_rounds.cc:70] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:18.239882 1308 leader_election.cc:290] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.239990 1166 pending_rounds.cc:77] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432194172030976 op_type: NO_OP noop_request { }
I20250625 07:52:18.240170 1166 raft_consensus.cc:2887] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:18.240288 1166 raft_consensus.cc:2270] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.240963 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76fe692067954856908f936ced332c18" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:18.241588 851 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.242655 1166 ts_tablet_manager.cc:1905] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.246423 590 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 90e5dd2a97f74363b43975b085da78af
W20250625 07:52:18.247828 986 leader_election.cc:343] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 23755f70822740e08ca6a5e5b5571458
I20250625 07:52:18.248203 747 tablet_service.cc:1515] Processing DeleteTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.250909 747 tablet_service.cc:1468] Processing CreateTablet for tablet fe630469fe7c4f4bac28277896eddc9c (DEFAULT_TABLE table=default.Table_bebb528a00504a7e80d68cec36278f45 [id=75a65f2d29cb4ccdb6555b7f1d0fe16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.252050 1163 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.252308 1163 ts_tablet_manager.cc:1397] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.134s user 0.000s sys 0.003s
W20250625 07:52:18.253973 985 leader_election.cc:343] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 23755f70822740e08ca6a5e5b5571458
I20250625 07:52:18.254238 985 leader_election.cc:304] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:18.255074 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:18.255481 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76fe692067954856908f936ced332c18" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:18.256033 850 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.256179 850 leader_election.cc:304] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:18.256451 1308 raft_consensus.cc:2747] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.256705 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_973a2ebd7372436296e18cdf46fc490f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.257361 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a3f6090fd2ea47c1a84f1e6b83a22af8" } new_table_name: "default.Table_ba10e2f968a543d4a5173431533d7e72" modify_external_catalogs: true
I20250625 07:52:18.259312 1485 raft_consensus.cc:2747] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:18.259330 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_973a2ebd7372436296e18cdf46fc490f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.261497 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 46389.4us
I20250625 07:52:18.262449 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e77df5e7b3447fdbdb1157e74cf8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.262202 1161 ts_tablet_manager.cc:1918] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.263372 1161 log.cc:1199] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/09b3830bd506443ab6f06ddd3c6b9974
I20250625 07:52:18.265869 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_83f5d615c53042dd888e74c415cc3caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.266135 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1770f6d5d1d4899a66ec23ab4eadc35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:18.266398 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83f5d615c53042dd888e74c415cc3caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.266979 1013 tablet_service.cc:1468] Processing CreateTablet for tablet fe630469fe7c4f4bac28277896eddc9c (DEFAULT_TABLE table=default.Table_bebb528a00504a7e80d68cec36278f45 [id=75a65f2d29cb4ccdb6555b7f1d0fe16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.267949 895 tablet_service.cc:1468] Processing CreateTablet for tablet fe630469fe7c4f4bac28277896eddc9c (DEFAULT_TABLE table=default.Table_bebb528a00504a7e80d68cec36278f45 [id=75a65f2d29cb4ccdb6555b7f1d0fe16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.268395 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe630469fe7c4f4bac28277896eddc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.271950 881 tablet_service.cc:1468] Processing CreateTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 (DEFAULT_TABLE table=default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.272358 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdbdd1e3b734463ab452bafecb75822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.276319 880 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfa1c4d4b6e4530892118189a5ab516 (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.276664 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfa1c4d4b6e4530892118189a5ab516. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.276741 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5de1492e483e47919ee5581ecee227be consensus state has no leader
I20250625 07:52:18.278462 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3728160a80af42f395da276c742fe44f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.278961 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3728160a80af42f395da276c742fe44f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.280640 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdad7dc1b27e4d87a747642a6d47230c consensus state has no leader
I20250625 07:52:18.286080 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a (DEFAULT_TABLE table=default.Table_7fc614d790ab4feaa0e3167c5de7cffd [id=93204c5397784d4fa49cd180005e4d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.287103 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfa1c4d4b6e4530892118189a5ab516 (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.288539 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe630469fe7c4f4bac28277896eddc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.290233 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 (DEFAULT_TABLE table=default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.290901 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfa1c4d4b6e4530892118189a5ab516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.290915 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3200657505f94bf7a15245040bb3b059" } new_table_name: "default.Table_b8f8397cf384480e8f65fcbf6cd5de9d" modify_external_catalogs: true
I20250625 07:52:18.291245 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdbdd1e3b734463ab452bafecb75822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.292374 1161 ts_tablet_manager.cc:1939] T 09b3830bd506443ab6f06ddd3c6b9974 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:18.300100 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 368) took 1087 ms. Trace:
W20250625 07:52:18.300545 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abb4d9dc2da7453d8336d5589dda3a74 consensus state has no leader
I20250625 07:52:18.302786 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_49c77ad274eb4800b6cafdeafc645b7d" } modify_external_catalogs: true
I20250625 07:52:18.303323 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_49c77ad274eb4800b6cafdeafc645b7d" } modify_external_catalogs: true
I20250625 07:52:18.303715 1163 raft_consensus.cc:357] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.303942 1163 raft_consensus.cc:383] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.304019 1163 raft_consensus.cc:738] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.304284 1163 consensus_queue.cc:260] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.304994 1163 ts_tablet_manager.cc:1428] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250625 07:52:18.305429 1163 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.307672 1163 tablet_bootstrap.cc:654] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.308357 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.254465 1166 ts_tablet_manager.cc:1918] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:18.312844 1166 log.cc:1199] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9e77df5e7b3447fdbdb1157e74cf8b6d
I20250625 07:52:18.313323 1166 ts_tablet_manager.cc:1939] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.314363 1166 tablet_replica.cc:331] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.314543 1166 raft_consensus.cc:2241] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.314752 1166 raft_consensus.cc:2270] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.255288 1162 tablet_bootstrap.cc:654] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:18.259742 713 leader_election.cc:343] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:18.316597 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 56828.6us
I20250625 07:52:18.267786 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 32bd7cdef96946e88683e457c2f26c9b with delete_type TABLET_DATA_DELETED (replaced by tablet 2db37ece7a964f47911fe4fc7434cd07) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.317618 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 (DEFAULT_TABLE table=default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc [id=f24c46ab73494d2b958f0fa750eadc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.318032 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cdbdd1e3b734463ab452bafecb75822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.318364 1017 tablet_service.cc:1468] Processing CreateTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 (DEFAULT_TABLE table=default.Table_d375845276c6425baf08c8e94b318afc [id=74a388127a4343ff94b26af12d3d1577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.319482 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.320116 1166 ts_tablet_manager.cc:1905] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.319540 1161 rpcz_store.cc:276] 0625 07:52:17.205123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.207641 (+ 2518us) service_pool.cc:225] Handling call
0625 07:52:18.154712 (+947071us) ts_tablet_manager.cc:1145] Deleting tablet 09b3830bd506443ab6f06ddd3c6b9974
0625 07:52:18.154724 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.163167 (+ 8443us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.258160 (+ 94993us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.292790 (+ 34630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":937372}
I20250625 07:52:18.321874 1161 tablet_replica.cc:331] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:18.322274 1161 raft_consensus.cc:2241] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.326261 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0bfa1c4d4b6e4530892118189a5ab516 (DEFAULT_TABLE table=default.Table_8e8ec360a4e6476db25022ee41cf9eba [id=e10e5f4032ce4e51b479b74f9193de5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.326735 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bfa1c4d4b6e4530892118189a5ab516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.329005 590 master_service.cc:948] ReplaceTablet: received request to replace tablet b377c8111c9f4ac2ba05e65694d611b9 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.330814 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 09b3830bd506443ab6f06ddd3c6b9974 (table default.Table_9b6b2d22f75c46b1bdf831fb7702f7c2 [id=246be34d5b41481ca587cf743e57f917]) successfully deleted
I20250625 07:52:18.330978 1308 raft_consensus.cc:491] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.331148 1308 raft_consensus.cc:513] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:18.331398 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 086c9321a70e44b5bb3ea419bb466fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c9321a70e44b5bb3ea419bb466fbe already in progress: creating tablet
I20250625 07:52:18.331885 1308 leader_election.cc:290] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.332433 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e6c2988baa14cbc8c878f79e942880d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:18.333031 851 leader_election.cc:343] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.333292 882 tablet_service.cc:1468] Processing CreateTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 (DEFAULT_TABLE table=default.Table_d375845276c6425baf08c8e94b318afc [id=74a388127a4343ff94b26af12d3d1577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.333523 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_458ec3bda21d4689bdd0cfa9ac3e6dc1" } modify_external_catalogs: true
I20250625 07:52:18.333696 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62dfb8f20f4caeaa79ea1e85d99930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.333716 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_458ec3bda21d4689bdd0cfa9ac3e6dc1" } modify_external_catalogs: true
I20250625 07:52:18.335654 1161 raft_consensus.cc:2270] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.337261 1161 ts_tablet_manager.cc:1905] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.337577 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e6c2988baa14cbc8c878f79e942880d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:18.338097 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62dfb8f20f4caeaa79ea1e85d99930. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.338888 850 leader_election.cc:343] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.339025 850 leader_election.cc:304] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:18.339320 1308 raft_consensus.cc:2747] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.340811 757 tablet_service.cc:1468] Processing CreateTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 (DEFAULT_TABLE table=default.Table_d375845276c6425baf08c8e94b318afc [id=74a388127a4343ff94b26af12d3d1577]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.341243 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62dfb8f20f4caeaa79ea1e85d99930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.344861 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (table default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]) successfully deleted
I20250625 07:52:18.344889 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_87717ced5eb64d5fb6553780090af598" } modify_external_catalogs: true
I20250625 07:52:18.345104 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_87717ced5eb64d5fb6553780090af598" } modify_external_catalogs: true
I20250625 07:52:18.345865 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 651c74bf9a9b47d6ab5a875a5c304783 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:18.355479 713 leader_election.cc:343] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.360715 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 49697a10cdac4c8ebb69f10f4c3437ec from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.362015 596 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 528df2e391cc42398e18e075876b180a
I20250625 07:52:18.364552 1166 ts_tablet_manager.cc:1918] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.368554 1158 tablet_bootstrap.cc:492] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:18.369716 1158 ts_tablet_manager.cc:1397] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.146s user 0.005s sys 0.008s
I20250625 07:52:18.370277 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d8883124b71c413c9ebf7ede2f2052f1
I20250625 07:52:18.370719 590 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ee3a8713a298486d8eb35eaaa6238200
I20250625 07:52:18.370600 1158 raft_consensus.cc:357] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.371026 1158 raft_consensus.cc:383] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.371129 1158 raft_consensus.cc:738] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.372125 1158 consensus_queue.cc:260] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.373548 1166 log.cc:1199] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c81396ee6ed7400baa30dcc5d71ba45d
I20250625 07:52:18.373907 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe630469fe7c4f4bac28277896eddc9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.375191 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 382) took 1009 ms. Trace:
I20250625 07:52:18.375347 1160 rpcz_store.cc:276] 0625 07:52:17.366173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.368904 (+ 2731us) service_pool.cc:225] Handling call
0625 07:52:17.934523 (+565619us) ts_tablet_manager.cc:1145] Deleting tablet 5f62b03db94d4ee3935866b5dd8bd53f
0625 07:52:17.934534 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:17.939363 (+ 4829us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.166435 (+227072us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.375152 (+208717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":565408}
I20250625 07:52:18.377338 1160 tablet_replica.cc:331] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:18.377565 1160 raft_consensus.cc:2241] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.377753 1160 raft_consensus.cc:2270] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.375017 1166 ts_tablet_manager.cc:1939] T c81396ee6ed7400baa30dcc5d71ba45d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.379534 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c81396ee6ed7400baa30dcc5d71ba45d (table default.Table_3aec9b6017944d5f8293424556b380d3 [id=975c0e95149a474b8e98ad4c4411b385]) successfully deleted
I20250625 07:52:18.380976 1166 tablet_replica.cc:331] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.381165 1166 raft_consensus.cc:2241] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.381315 1166 raft_consensus.cc:2270] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.383386 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5f62b03db94d4ee3935866b5dd8bd53f (table default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]) successfully deleted
W20250625 07:52:18.383646 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 086c9321a70e44b5bb3ea419bb466fbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 086c9321a70e44b5bb3ea419bb466fbe already in progress: creating tablet
W20250625 07:52:18.383874 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 541c983d64f348539979f74199f79e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541c983d64f348539979f74199f79e59 already in progress: creating tablet
W20250625 07:52:18.384080 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 110fe299c7c44df69f459da72a1138a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110fe299c7c44df69f459da72a1138a5 already in progress: creating tablet
W20250625 07:52:18.384299 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3b9491eb269948a29888a7301ea75d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b9491eb269948a29888a7301ea75d42 already in progress: creating tablet
W20250625 07:52:18.384498 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ba0735d576f947d395bee8d1b865f2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba0735d576f947d395bee8d1b865f2fd already in progress: creating tablet
W20250625 07:52:18.384738 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 already in progress: creating tablet
W20250625 07:52:18.384934 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2d8d3300befc4f419070ae84725ee96a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8d3300befc4f419070ae84725ee96a already in progress: creating tablet
W20250625 07:52:18.385149 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 743123ffcf8240dc82bea7178559b7ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 743123ffcf8240dc82bea7178559b7ea already in progress: creating tablet
W20250625 07:52:18.385377 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af43b9ceeb4f2b812ee251ea8545ce already in progress: creating tablet
W20250625 07:52:18.385746 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29eaa8d93e5b4d1d8ed147b60c7da859 already in progress: creating tablet
W20250625 07:52:18.385938 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bcfe7c3c54741fbb0aca69650fc8eee already in progress: creating tablet
I20250625 07:52:18.386793 1158 ts_tablet_manager.cc:1428] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:18.386865 1160 ts_tablet_manager.cc:1905] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.387207 1158 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:18.388454 1166 ts_tablet_manager.cc:1905] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.389474 1162 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.389611 1162 ts_tablet_manager.cc:1397] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.152s user 0.002s sys 0.009s
I20250625 07:52:18.390439 1162 raft_consensus.cc:357] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.390604 1162 raft_consensus.cc:383] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.390674 1162 raft_consensus.cc:738] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.390887 1162 consensus_queue.cc:260] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.391441 1162 ts_tablet_manager.cc:1428] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:18.393515 1162 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.394372 1158 tablet_bootstrap.cc:654] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.401778 1161 ts_tablet_manager.cc:1918] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.401942 1161 log.cc:1199] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3338d1c708b4421c8501740da89afec5
I20250625 07:52:18.401050 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.403658 1160 ts_tablet_manager.cc:1918] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.406591 1162 tablet_bootstrap.cc:654] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.406759 1161 ts_tablet_manager.cc:1939] T 3338d1c708b4421c8501740da89afec5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:18.409227 1158 tablet_bootstrap.cc:492] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:18.409375 1158 ts_tablet_manager.cc:1397] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250625 07:52:18.410279 1158 raft_consensus.cc:357] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.410475 1158 raft_consensus.cc:383] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.410578 1158 raft_consensus.cc:738] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.410817 1158 consensus_queue.cc:260] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.410840 1161 tablet_replica.cc:331] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:18.411487 1158 ts_tablet_manager.cc:1428] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:18.412102 1158 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:18.413563 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6c9346951d87447e84acea414f3cc804"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.414162 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9346951d87447e84acea414f3cc804 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.415026 1161 raft_consensus.cc:2241] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.415704 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3338d1c708b4421c8501740da89afec5 (table default.Table_8eab5a30488e4505bfd1a2a7149fe51d [id=8369867cfe5c4114aa69916d8b10f00f]) successfully deleted
W20250625 07:52:18.415979 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 110fe299c7c44df69f459da72a1138a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 110fe299c7c44df69f459da72a1138a5 already in progress: creating tablet
W20250625 07:52:18.416220 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3b9491eb269948a29888a7301ea75d42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b9491eb269948a29888a7301ea75d42 already in progress: creating tablet
W20250625 07:52:18.416435 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 541c983d64f348539979f74199f79e59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 541c983d64f348539979f74199f79e59 already in progress: creating tablet
W20250625 07:52:18.416626 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2d8d3300befc4f419070ae84725ee96a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d8d3300befc4f419070ae84725ee96a already in progress: creating tablet
W20250625 07:52:18.416829 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 already in progress: creating tablet
W20250625 07:52:18.417057 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ba0735d576f947d395bee8d1b865f2fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba0735d576f947d395bee8d1b865f2fd already in progress: creating tablet
I20250625 07:52:18.422686 1161 raft_consensus.cc:2270] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.422868 1166 ts_tablet_manager.cc:1918] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.423022 1166 log.cc:1199] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/32bd7cdef96946e88683e457c2f26c9b
I20250625 07:52:18.423414 1166 ts_tablet_manager.cc:1939] T 32bd7cdef96946e88683e457c2f26c9b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.423972 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 9a90bb52358f4c38a540ad025e9e0430 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.424407 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 32bd7cdef96946e88683e457c2f26c9b (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]) successfully deleted
I20250625 07:52:18.424770 1162 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.424998 1162 ts_tablet_manager.cc:1397] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250625 07:52:18.425745 1162 raft_consensus.cc:357] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.425932 1162 raft_consensus.cc:383] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.426004 1162 raft_consensus.cc:738] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.426218 1162 consensus_queue.cc:260] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.426680 1162 ts_tablet_manager.cc:1428] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:18.427029 1162 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.433008 747 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.434655 1166 tablet_replica.cc:331] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.434859 1166 raft_consensus.cc:2241] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.435031 1166 raft_consensus.cc:2270] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.436293 1162 tablet_bootstrap.cc:654] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.438740 1166 ts_tablet_manager.cc:1905] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.442350 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0352225d645c408ba8c74273ee5f56e6 (DEFAULT_TABLE table=default.Table_f98ca9d11df0483887149bfbb92885be [id=9c2f61bc035a4a26a4c079902b04259a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:18.442494 1485 raft_consensus.cc:668] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250625 07:52:18.442517 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1973678168a84875b747897f08dd1bad (DEFAULT_TABLE table=default.Table_83f5d615c53042dd888e74c415cc3caf [id=1f730d2bc75b4fab86e26a382a94ce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.442806 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0352225d645c408ba8c74273ee5f56e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.443063 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1973678168a84875b747897f08dd1bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.444728 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c0f428f2b54479a8e0bd1f36ef2dede" } modify_external_catalogs: true
I20250625 07:52:18.444912 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_06060aabdbd54758b16137634d989c0c" } new_table_name: "default.Table_1444a3af50a54dfe9739e5c255dc27a4" modify_external_catalogs: true
I20250625 07:52:18.444959 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c0f428f2b54479a8e0bd1f36ef2dede" } modify_external_catalogs: true
I20250625 07:52:18.446533 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_69159065e6254990b2bebfab94d2fd0b" } new_table_name: "default.Table_2fb37c46511c4963a2b384669ec6331c" modify_external_catalogs: true
I20250625 07:52:18.448784 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8df215ea93ca49c391a3c3526be6fc22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.449477 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df215ea93ca49c391a3c3526be6fc22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.449599 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a1adb2364ae94c0aa62bc2d84fc9aded" } modify_external_catalogs: true
I20250625 07:52:18.449805 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a1adb2364ae94c0aa62bc2d84fc9aded" } modify_external_catalogs: true
I20250625 07:52:18.451112 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_41047b79cb784bb19f458dbce4573ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.451592 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41047b79cb784bb19f458dbce4573ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.455132 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 (DEFAULT_TABLE table=default.Table_9f2673ddf3e34a969d58c4514580bdcd [id=2aad9533753e4ff9b9c0bab636a8f85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.455242 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 (DEFAULT_TABLE table=default.Table_380bf0b7bfc248f7b340c14b230c4db7 [id=f129efbf6ddb47d1a59e7f511637654a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.455955 757 tablet_service.cc:1468] Processing CreateTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 (DEFAULT_TABLE table=default.Table_380bf0b7bfc248f7b340c14b230c4db7 [id=f129efbf6ddb47d1a59e7f511637654a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.456383 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cd482ceec244c2bed2b116d5c14d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.405076 1163 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.421021 915 raft_consensus.cc:2466] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:18.457020 1163 ts_tablet_manager.cc:1397] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.152s user 0.004s sys 0.000s
I20250625 07:52:18.445866 881 tablet_service.cc:1468] Processing CreateTablet for tablet c858e286fbba450e81c6cc55f0f2b613 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.457397 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c858e286fbba450e81c6cc55f0f2b613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.457371 713 leader_election.cc:304] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:18.446636 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1973678168a84875b747897f08dd1bad (DEFAULT_TABLE table=default.Table_83f5d615c53042dd888e74c415cc3caf [id=1f730d2bc75b4fab86e26a382a94ce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.457872 1333 raft_consensus.cc:2802] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:18.457937 1163 raft_consensus.cc:357] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.458048 1333 raft_consensus.cc:695] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:18.458323 1333 consensus_queue.cc:237] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.447321 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0352225d645c408ba8c74273ee5f56e6 (DEFAULT_TABLE table=default.Table_f98ca9d11df0483887149bfbb92885be [id=9c2f61bc035a4a26a4c079902b04259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.458058 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1973678168a84875b747897f08dd1bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.458133 1163 raft_consensus.cc:383] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.460858 1163 raft_consensus.cc:738] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.461092 1163 consensus_queue.cc:260] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.461652 1163 ts_tablet_manager.cc:1428] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:18.463557 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0352225d645c408ba8c74273ee5f56e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.464841 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 0352225d645c408ba8c74273ee5f56e6 (DEFAULT_TABLE table=default.Table_f98ca9d11df0483887149bfbb92885be [id=9c2f61bc035a4a26a4c079902b04259a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.465127 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a396bde4b5964697ad0090eab777ba0e (DEFAULT_TABLE table=default.Table_e1eb60c1e8c146a596391600eaffc9d7 [id=a0beb6150680402fbd0035acae23bcf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.465564 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1973678168a84875b747897f08dd1bad (DEFAULT_TABLE table=default.Table_83f5d615c53042dd888e74c415cc3caf [id=1f730d2bc75b4fab86e26a382a94ce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.465963 1014 tablet_service.cc:1468] Processing CreateTablet for tablet c858e286fbba450e81c6cc55f0f2b613 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.466132 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fdb6a689ce6440ab7037484e2b9cdc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.466895 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e66f393d52d48938a989a0c7104879c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.467478 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e66f393d52d48938a989a0c7104879c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.448058 880 tablet_service.cc:1468] Processing CreateTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 (DEFAULT_TABLE table=default.Table_380bf0b7bfc248f7b340c14b230c4db7 [id=f129efbf6ddb47d1a59e7f511637654a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.469890 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cd482ceec244c2bed2b116d5c14d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.471964 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.448799 890 tablet_service.cc:1468] Processing CreateTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 (DEFAULT_TABLE table=default.Table_9f2673ddf3e34a969d58c4514580bdcd [id=2aad9533753e4ff9b9c0bab636a8f85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.475258 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bfcc2b20ba4d0b83641988f2c0ee35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.475224 893 tablet_service.cc:1468] Processing CreateTablet for tablet 034903d699584feb91f35cbb2f62701a (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.475574 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034903d699584feb91f35cbb2f62701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.449702 885 tablet_service.cc:1468] Processing CreateTablet for tablet 065822c50586488785f7d9d53841a4dd (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.476040 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 065822c50586488785f7d9d53841a4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.479660 894 tablet_service.cc:1468] Processing CreateTablet for tablet d7ed91509ba94562a9207c7d7c0d3e2b (DEFAULT_TABLE table=default.Table_1c370799a55b474080d93cb980cf5f39 [id=1353496ce19c4b14bdcc41400f3e9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.450428 889 tablet_service.cc:1468] Processing CreateTablet for tablet a396bde4b5964697ad0090eab777ba0e (DEFAULT_TABLE table=default.Table_e1eb60c1e8c146a596391600eaffc9d7 [id=a0beb6150680402fbd0035acae23bcf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.451398 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c (DEFAULT_TABLE table=default.Table_c121aa9cbde14c6bb4a4bcb574484208 [id=b6a77883382d4a6f8ff92139c24ba3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.480533 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ed91509ba94562a9207c7d7c0d3e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.452793 884 tablet_service.cc:1468] Processing CreateTablet for tablet 486463a8140c4e769cc79870d2acff78 (DEFAULT_TABLE table=default.Table_5e910ce70a934be2be117a115fea3380 [id=1ecc6b62c7c140f1bff20d234b427666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.456019 886 tablet_service.cc:1468] Processing CreateTablet for tablet e10149f4ae0645ba88508ca51ae9f77a (DEFAULT_TABLE table=default.Table_8c340466a42e47dead226f74c8176f49 [id=b086d131091348dcbd465e24e269721e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.481084 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486463a8140c4e769cc79870d2acff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.481710 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e26b4b26dfe434ba65119e99f41bb8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.482288 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e26b4b26dfe434ba65119e99f41bb8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.484840 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10149f4ae0645ba88508ca51ae9f77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.485437 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0b9db8ced3e74eecb4fc928446f87bc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.485998 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b9db8ced3e74eecb4fc928446f87bc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.486896 1166 ts_tablet_manager.cc:1918] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.487036 1166 log.cc:1199] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:18.487407 1166 ts_tablet_manager.cc:1939] T 6b1eaf0266c04873b185c1c0c2da201c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.488847 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3488f79d09054f6dbfc6b54c1dee0d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.489230 744 tablet_service.cc:1468] Processing CreateTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 (DEFAULT_TABLE table=default.Table_9f2673ddf3e34a969d58c4514580bdcd [id=2aad9533753e4ff9b9c0bab636a8f85b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.489627 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bfcc2b20ba4d0b83641988f2c0ee35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.490180 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:18.490571 595 catalog_manager.cc:6652] ReplaceTablet: tablet 9a90bb52358f4c38a540ad025e9e0430 deleted and replaced by tablet d0a75906385744e5ba6938f451a2d7ec
I20250625 07:52:18.490928 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_66abf1d5bf9b498aacef39657e91e79c" } new_table_name: "default.Table_4d062bccb2764b73a210a359c367f0cb" modify_external_catalogs: true
I20250625 07:52:18.492830 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a396bde4b5964697ad0090eab777ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.495604 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 547b758c2e6c4558bb232f14b37aa81a (table default.Table_0c0f428f2b54479a8e0bd1f36ef2dede [id=83522331e5df4bafaa65c81be08c7525], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 547b758c2e6c4558bb232f14b37aa81a consensus state has no leader
I20250625 07:52:18.496205 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6b1eaf0266c04873b185c1c0c2da201c (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
I20250625 07:52:18.496407 745 tablet_service.cc:1468] Processing CreateTablet for tablet 065822c50586488785f7d9d53841a4dd (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.496843 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 065822c50586488785f7d9d53841a4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.497774 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.498023 1163 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.499747 1163 tablet_bootstrap.cc:654] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.500506 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b167db2ea8944d13b34adbb44e815b62" } new_table_name: "default.Table_bb831997b8cf4b99a7779a4a6b23a238" modify_external_catalogs: true
I20250625 07:52:18.501062 743 tablet_service.cc:1468] Processing CreateTablet for tablet a396bde4b5964697ad0090eab777ba0e (DEFAULT_TABLE table=default.Table_e1eb60c1e8c146a596391600eaffc9d7 [id=a0beb6150680402fbd0035acae23bcf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.501456 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a396bde4b5964697ad0090eab777ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.502805 894 tablet_service.cc:1468] Processing CreateTablet for tablet d6f04229a1d8462c818d589f9f4eb712 (DEFAULT_TABLE table=default.Table_eea0319c78844065b1dc101bb7ff333c [id=e875baa90eb9498b9d5a99e2e2b86942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.503217 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f04229a1d8462c818d589f9f4eb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.505117 750 tablet_service.cc:1468] Processing CreateTablet for tablet 486463a8140c4e769cc79870d2acff78 (DEFAULT_TABLE table=default.Table_5e910ce70a934be2be117a115fea3380 [id=1ecc6b62c7c140f1bff20d234b427666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.505492 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486463a8140c4e769cc79870d2acff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.509052 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c (DEFAULT_TABLE table=default.Table_c121aa9cbde14c6bb4a4bcb574484208 [id=b6a77883382d4a6f8ff92139c24ba3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.509418 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3488f79d09054f6dbfc6b54c1dee0d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.512280 758 tablet_service.cc:1468] Processing CreateTablet for tablet e10149f4ae0645ba88508ca51ae9f77a (DEFAULT_TABLE table=default.Table_8c340466a42e47dead226f74c8176f49 [id=b086d131091348dcbd465e24e269721e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.512629 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10149f4ae0645ba88508ca51ae9f77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.515993 747 tablet_service.cc:1468] Processing CreateTablet for tablet c858e286fbba450e81c6cc55f0f2b613 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.516983 753 tablet_service.cc:1468] Processing CreateTablet for tablet 034903d699584feb91f35cbb2f62701a (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.517315 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034903d699584feb91f35cbb2f62701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.518394 755 tablet_service.cc:1468] Processing CreateTablet for tablet d7ed91509ba94562a9207c7d7c0d3e2b (DEFAULT_TABLE table=default.Table_1c370799a55b474080d93cb980cf5f39 [id=1353496ce19c4b14bdcc41400f3e9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.518703 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ed91509ba94562a9207c7d7c0d3e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.519491 888 tablet_service.cc:1468] Processing CreateTablet for tablet ea2b581d722d4a8283005b37106f5b51 (DEFAULT_TABLE table=default.Table_e362ea24c1794efbb1c667870df19840 [id=c5262794b5744c40870f10fce9e86e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.519888 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2b581d722d4a8283005b37106f5b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.522544 1474 raft_consensus.cc:491] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.522699 1474 raft_consensus.cc:513] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.523573 1474 leader_election.cc:290] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:18.524984 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6c0834adf1748c6a67805d26b973171 consensus state has no leader
W20250625 07:52:18.525324 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02ec8c9e6340497abe802c7345f42921 consensus state has no leader
I20250625 07:52:18.525771 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e35c55dc42af4bb28966deeb87d520c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.526404 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e35c55dc42af4bb28966deeb87d520c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.527447 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aab39ebeac1742cda147feb57cf092d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.527900 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 547b758c2e6c4558bb232f14b37aa81a
W20250625 07:52:18.528084 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aab39ebeac1742cda147feb57cf092d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.529206 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_15745c1b68f54870b34b4ced0001369f" } new_table_name: "default.Table_9cacaaab2fce44ca9bb02a60b6edf5ff" modify_external_catalogs: true
I20250625 07:52:18.529891 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e181a3f1ae2c49349718d842c166b1e6
I20250625 07:52:18.530546 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7e8eb7e9baf5497ca408e7592b57e4dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.531013 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e8eb7e9baf5497ca408e7592b57e4dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.528628 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_71387ac73a3149429c4f14fe3995a852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.531662 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71387ac73a3149429c4f14fe3995a852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.539448 1160 log.cc:1199] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/ebb943df93284913b6a520e099399322
I20250625 07:52:18.539862 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.540179 756 tablet_service.cc:1468] Processing CreateTablet for tablet d6f04229a1d8462c818d589f9f4eb712 (DEFAULT_TABLE table=default.Table_eea0319c78844065b1dc101bb7ff333c [id=e875baa90eb9498b9d5a99e2e2b86942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.540436 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 065822c50586488785f7d9d53841a4dd (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.540549 1166 tablet_replica.cc:331] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:18.540882 1166 raft_consensus.cc:2241] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.541136 1166 raft_consensus.cc:2270] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.541112 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:18.541328 915 raft_consensus.cc:2466] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:18.541869 713 leader_election.cc:304] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:18.544116 594 catalog_manager.cc:5582] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:18.544564 594 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]). Expected version 2 got 0
I20250625 07:52:18.546679 886 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96cdddd3443b8b67f57f4d6c4846e (DEFAULT_TABLE table=default.Table_7e6dadd5349f42d08e3e43a15ca0c845 [id=1082e4c0af594912bd6b02bd57557cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.548569 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96cdddd3443b8b67f57f4d6c4846e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.549229 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 486463a8140c4e769cc79870d2acff78 (DEFAULT_TABLE table=default.Table_5e910ce70a934be2be117a115fea3380 [id=1ecc6b62c7c140f1bff20d234b427666]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.549646 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.550184 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c (DEFAULT_TABLE table=default.Table_c121aa9cbde14c6bb4a4bcb574484208 [id=b6a77883382d4a6f8ff92139c24ba3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.555037 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d6f04229a1d8462c818d589f9f4eb712 (DEFAULT_TABLE table=default.Table_eea0319c78844065b1dc101bb7ff333c [id=e875baa90eb9498b9d5a99e2e2b86942]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.556609 1028 tablet_service.cc:1468] Processing CreateTablet for tablet e10149f4ae0645ba88508ca51ae9f77a (DEFAULT_TABLE table=default.Table_8c340466a42e47dead226f74c8176f49 [id=b086d131091348dcbd465e24e269721e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.557408 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d7ed91509ba94562a9207c7d7c0d3e2b (DEFAULT_TABLE table=default.Table_1c370799a55b474080d93cb980cf5f39 [id=1353496ce19c4b14bdcc41400f3e9844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.558563 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7d930a09f5e542aab943a51072f0c626 (DEFAULT_TABLE table=default.Table_9f7e1cd61b2d4a87b378410741a2a5f9 [id=d321002cd47242a68bda2f0a7236c9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.558670 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 034903d699584feb91f35cbb2f62701a (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.558943 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e181a3f1ae2c49349718d842c166b1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.558945 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 7d930a09f5e542aab943a51072f0c626 (DEFAULT_TABLE table=default.Table_9f7e1cd61b2d4a87b378410741a2a5f9 [id=d321002cd47242a68bda2f0a7236c9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.559855 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9f31104720614b028e373ae1500c31cc (DEFAULT_TABLE table=default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289 [id=82f3c359f3be4716987d150b7251db98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.560523 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08cd482ceec244c2bed2b116d5c14d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.560642 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:18.560844 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bfcc2b20ba4d0b83641988f2c0ee35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.561776 884 tablet_service.cc:1468] Processing CreateTablet for tablet a40dacaa394747aaa100a0527e14eed0 (DEFAULT_TABLE table=default.Table_f801c447326646d1ab93ac87fff7843a [id=cf2c5d2f59bc462bb1fe081f619e5841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.563395 1029 tablet_service.cc:1468] Processing CreateTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 (DEFAULT_TABLE table=default.Table_c62e57a0d535414a93d82dca7dd477ba [id=1dbec332cb8a45eb8b6114bd1b9bdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.564940 883 tablet_service.cc:1468] Processing CreateTablet for tablet a6c0834adf1748c6a67805d26b973171 (DEFAULT_TABLE table=default.Table_1444a3af50a54dfe9739e5c255dc27a4 [id=8e2d9c1cd8cd4b149d3be543ab086667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.566445 889 tablet_service.cc:1468] Processing CreateTablet for tablet c82b840442ec4cb4993636183812ed22 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.564579 1161 ts_tablet_manager.cc:1905] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.568442 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5a458f32a2864c59a2c926d7760d2f13" } modify_external_catalogs: true
I20250625 07:52:18.568647 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5a458f32a2864c59a2c926d7760d2f13" } modify_external_catalogs: true
I20250625 07:52:18.574630 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1437af2855fc49d6b981d94261019399" } modify_external_catalogs: true
I20250625 07:52:18.574854 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1437af2855fc49d6b981d94261019399" } modify_external_catalogs: true
I20250625 07:52:18.578635 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.582211 1160 ts_tablet_manager.cc:1939] T ebb943df93284913b6a520e099399322 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:18.583900 895 tablet_service.cc:1468] Processing CreateTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 (DEFAULT_TABLE table=default.Table_5a458f32a2864c59a2c926d7760d2f13 [id=9e7bb49609f34ef4adadd800b22843e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.589946 888 tablet_service.cc:1468] Processing CreateTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 (DEFAULT_TABLE table=default.Table_c62e57a0d535414a93d82dca7dd477ba [id=1dbec332cb8a45eb8b6114bd1b9bdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.590296 1485 raft_consensus.cc:491] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.590497 1485 raft_consensus.cc:513] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.589931 1011 tablet_service.cc:1468] Processing CreateTablet for tablet ea2b581d722d4a8283005b37106f5b51 (DEFAULT_TABLE table=default.Table_e362ea24c1794efbb1c667870df19840 [id=c5262794b5744c40870f10fce9e86e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.592236 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d930a09f5e542aab943a51072f0c626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.593200 1373 raft_consensus.cc:491] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.594561 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96cdddd3443b8b67f57f4d6c4846e (DEFAULT_TABLE table=default.Table_7e6dadd5349f42d08e3e43a15ca0c845 [id=1082e4c0af594912bd6b02bd57557cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.595103 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.595453 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet e181a3f1ae2c49349718d842c166b1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.596164 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 (DEFAULT_TABLE table=default.Table_5a458f32a2864c59a2c926d7760d2f13 [id=9e7bb49609f34ef4adadd800b22843e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.596506 1373 raft_consensus.cc:513] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.597611 1373 leader_election.cc:290] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:18.597975 1485 leader_election.cc:290] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:18.599838 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c0834adf1748c6a67805d26b973171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.602241 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c82b840442ec4cb4993636183812ed22 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.602717 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d930a09f5e542aab943a51072f0c626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.603080 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1973678168a84875b747897f08dd1bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.603296 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82b840442ec4cb4993636183812ed22. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.603144 985 leader_election.cc:343] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:18.604849 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40dacaa394747aaa100a0527e14eed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.608847 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82b840442ec4cb4993636183812ed22. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.608999 985 leader_election.cc:343] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b7139335c7124ab79bc2d0a251984199
I20250625 07:52:18.609280 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96cdddd3443b8b67f57f4d6c4846e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.609526 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486463a8140c4e769cc79870d2acff78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.612341 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2b581d722d4a8283005b37106f5b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.612836 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8694f09dcf47ab9f5fc8854c21c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.612838 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dddbc1200acf4e7fa93ef85b398d5832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.616854 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8694f09dcf47ab9f5fc8854c21c331. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.620982 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f31104720614b028e373ae1500c31cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.621397 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a396bde4b5964697ad0090eab777ba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.622241 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 065822c50586488785f7d9d53841a4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.628842 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10149f4ae0645ba88508ca51ae9f77a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.629813 1160 tablet_replica.cc:331] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:18.630188 1160 raft_consensus.cc:2241] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.630484 1160 raft_consensus.cc:2270] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.632797 1485 raft_consensus.cc:491] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.632872 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0352225d645c408ba8c74273ee5f56e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.632972 1485 raft_consensus.cc:513] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.633639 1163 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.633671 1485 leader_election.cc:290] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:18.633944 1163 ts_tablet_manager.cc:1397] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.005s
I20250625 07:52:18.634186 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dddbc1200acf4e7fa93ef85b398d5832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.635347 1163 raft_consensus.cc:357] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.635687 1163 raft_consensus.cc:383] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.635819 1163 raft_consensus.cc:738] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.636276 1163 consensus_queue.cc:260] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.638634 1163 ts_tablet_manager.cc:1428] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:18.640336 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034903d699584feb91f35cbb2f62701a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.640460 1163 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.641583 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:18.644840 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ed91509ba94562a9207c7d7c0d3e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.648937 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c858e286fbba450e81c6cc55f0f2b613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.651655 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a40dacaa394747aaa100a0527e14eed0 (DEFAULT_TABLE table=default.Table_f801c447326646d1ab93ac87fff7843a [id=cf2c5d2f59bc462bb1fe081f619e5841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.652071 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40dacaa394747aaa100a0527e14eed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.654836 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3488f79d09054f6dbfc6b54c1dee0d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.655795 1160 ts_tablet_manager.cc:1905] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.656455 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f04229a1d8462c818d589f9f4eb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.677340 1166 ts_tablet_manager.cc:1905] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:18.677395 713 leader_election.cc:343] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.677328 1333 raft_consensus.cc:2802] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:18.678372 1333 raft_consensus.cc:491] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:18.678546 1333 raft_consensus.cc:3058] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.678647 1474 raft_consensus.cc:491] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.678820 1474 raft_consensus.cc:513] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.679419 1474 leader_election.cc:290] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.683336 1163 tablet_bootstrap.cc:654] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.684762 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:18.684965 915 raft_consensus.cc:2466] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:18.685401 713 leader_election.cc:304] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:18.685698 1474 raft_consensus.cc:2802] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:18.685786 1474 raft_consensus.cc:491] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:18.685882 1474 raft_consensus.cc:3058] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.684443 1333 raft_consensus.cc:513] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.689050 1474 raft_consensus.cc:513] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.689649 1474 leader_election.cc:290] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.689673 1333 leader_election.cc:290] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:18.690589 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:18.690802 915 raft_consensus.cc:3058] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.690754 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:18.691097 914 raft_consensus.cc:3058] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:18.694038 915 raft_consensus.cc:2466] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:18.695709 914 raft_consensus.cc:2466] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:18.695891 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet ebb943df93284913b6a520e099399322 (table default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]) successfully deleted
W20250625 07:52:18.696192 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af43b9ceeb4f2b812ee251ea8545ce already in progress: creating tablet
W20250625 07:52:18.696480 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bcfe7c3c54741fbb0aca69650fc8eee already in progress: creating tablet
W20250625 07:52:18.696729 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b2bb060cdd9a48688967989d7491cfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bb060cdd9a48688967989d7491cfc1 already in progress: creating tablet
I20250625 07:52:18.696993 713 leader_election.cc:304] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:18.697399 713 leader_election.cc:304] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:18.697690 1474 raft_consensus.cc:2802] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:18.698025 1333 raft_consensus.cc:2802] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:18.698138 1474 raft_consensus.cc:695] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:18.698423 1474 consensus_queue.cc:237] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.703054 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c858e286fbba450e81c6cc55f0f2b613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.703374 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f04229a1d8462c818d589f9f4eb712. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.704617 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9f31104720614b028e373ae1500c31cc (DEFAULT_TABLE table=default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289 [id=82f3c359f3be4716987d150b7251db98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.708585 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f31104720614b028e373ae1500c31cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.711418 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9f31104720614b028e373ae1500c31cc (DEFAULT_TABLE table=default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289 [id=82f3c359f3be4716987d150b7251db98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.711979 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f31104720614b028e373ae1500c31cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.716226 746 tablet_service.cc:1468] Processing CreateTablet for tablet ea2b581d722d4a8283005b37106f5b51 (DEFAULT_TABLE table=default.Table_e362ea24c1794efbb1c667870df19840 [id=c5262794b5744c40870f10fce9e86e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.719648 1160 ts_tablet_manager.cc:1918] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.722581 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea2b581d722d4a8283005b37106f5b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.726063 1160 log.cc:1199] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/49697a10cdac4c8ebb69f10f4c3437ec
I20250625 07:52:18.726636 1160 ts_tablet_manager.cc:1939] T 49697a10cdac4c8ebb69f10f4c3437ec P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:18.727137 986 leader_election.cc:343] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:18.727245 1485 raft_consensus.cc:491] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:18.729461 1160 tablet_replica.cc:331] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:18.729678 1160 raft_consensus.cc:2241] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:18.729841 1160 raft_consensus.cc:2270] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:18.730424 1160 ts_tablet_manager.cc:1905] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:18.737259 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7d930a09f5e542aab943a51072f0c626 (DEFAULT_TABLE table=default.Table_9f7e1cd61b2d4a87b378410741a2a5f9 [id=d321002cd47242a68bda2f0a7236c9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.738438 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d930a09f5e542aab943a51072f0c626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.739943 1333 raft_consensus.cc:695] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:18.743397 1333 consensus_queue.cc:237] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.745675 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.746157 758 tablet_service.cc:1468] Processing CreateTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 (DEFAULT_TABLE table=default.Table_c62e57a0d535414a93d82dca7dd477ba [id=1dbec332cb8a45eb8b6114bd1b9bdd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.746294 750 tablet_service.cc:1468] Processing CreateTablet for tablet c82b840442ec4cb4993636183812ed22 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.746582 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dddbc1200acf4e7fa93ef85b398d5832. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.746853 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c82b840442ec4cb4993636183812ed22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.747058 753 tablet_service.cc:1515] Processing DeleteTablet for tablet e181a3f1ae2c49349718d842c166b1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:18.747766 754 tablet_service.cc:1468] Processing CreateTablet for tablet a6c0834adf1748c6a67805d26b973171 (DEFAULT_TABLE table=default.Table_1444a3af50a54dfe9739e5c255dc27a4 [id=8e2d9c1cd8cd4b149d3be543ab086667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.747886 755 tablet_service.cc:1468] Processing CreateTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 (DEFAULT_TABLE table=default.Table_5a458f32a2864c59a2c926d7760d2f13 [id=9e7bb49609f34ef4adadd800b22843e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.748154 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c0834adf1748c6a67805d26b973171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.748247 1160 ts_tablet_manager.cc:1918] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.748374 1160 log.cc:1199] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9e77df5e7b3447fdbdb1157e74cf8b6d
I20250625 07:52:18.748574 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8694f09dcf47ab9f5fc8854c21c331. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.750726 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 54033.7us
W20250625 07:52:18.751014 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9418b81f010d4f369d3409e9364bc5fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9418b81f010d4f369d3409e9364bc5fa already in progress: creating tablet
W20250625 07:52:18.751250 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b5f917cfd2384a72920847f8b89358b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f917cfd2384a72920847f8b89358b9 already in progress: creating tablet
W20250625 07:52:18.751446 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9cc1842f257d409ba0a56873f582f023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cc1842f257d409ba0a56873f582f023 already in progress: creating tablet
I20250625 07:52:18.751945 757 tablet_service.cc:1468] Processing CreateTablet for tablet a40dacaa394747aaa100a0527e14eed0 (DEFAULT_TABLE table=default.Table_f801c447326646d1ab93ac87fff7843a [id=cf2c5d2f59bc462bb1fe081f619e5841]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.752372 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 49697a10cdac4c8ebb69f10f4c3437ec (table default.Table_d7220ef926924a20949121c91fa08801 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:18.746598 745 tablet_service.cc:1468] Processing CreateTablet for tablet d1c96cdddd3443b8b67f57f4d6c4846e (DEFAULT_TABLE table=default.Table_7e6dadd5349f42d08e3e43a15ca0c845 [id=1082e4c0af594912bd6b02bd57557cf0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:18.752588 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a33b7cab19244ec08d3821b6cc4c072d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a33b7cab19244ec08d3821b6cc4c072d already in progress: creating tablet
I20250625 07:52:18.752889 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40dacaa394747aaa100a0527e14eed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.755725 753 tablet_service.cc:1468] Processing CreateTablet for tablet e181a3f1ae2c49349718d842c166b1e6 (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.753815 894 tablet_service.cc:1468] Processing CreateTablet for tablet e181a3f1ae2c49349718d842c166b1e6 (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.758069 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e181a3f1ae2c49349718d842c166b1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.754264 895 tablet_service.cc:1468] Processing CreateTablet for tablet 34544157495f4e14a7978e0eec70b522 (DEFAULT_TABLE table=default.Table_d057d5343c334e9e92d5d9d977f90e6a [id=04e37ec20f144d578c85368b35713aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.759459 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34544157495f4e14a7978e0eec70b522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.754604 886 tablet_service.cc:1468] Processing CreateTablet for tablet bee1c7f7746a4d6fb763aedadc861421 (DEFAULT_TABLE table=default.Table_3728160a80af42f395da276c742fe44f [id=bbe5def298bc4175997fb660f1bdf381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.763432 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee1c7f7746a4d6fb763aedadc861421. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.765048 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4b87309b76f4f0e9dde120d9f755c44 consensus state has no leader
I20250625 07:52:18.766276 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:18.768080 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_224bc28d99ba48a09163db6d3106dfc7" } modify_external_catalogs: true
I20250625 07:52:18.754878 888 tablet_service.cc:1468] Processing CreateTablet for tablet 26db0effa17b4cacb1450a01f9631ddf (DEFAULT_TABLE table=default.Table_f00b99b9279946bab7c178245d31c414 [id=5b100b36de77408ebb66b9d7cd64afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.769176 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db0effa17b4cacb1450a01f9631ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.769229 1485 raft_consensus.cc:513] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:18.758317 749 tablet_service.cc:1468] Processing CreateTablet for tablet 34544157495f4e14a7978e0eec70b522 (DEFAULT_TABLE table=default.Table_d057d5343c334e9e92d5d9d977f90e6a [id=04e37ec20f144d578c85368b35713aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.768273 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_224bc28d99ba48a09163db6d3106dfc7" } modify_external_catalogs: true
I20250625 07:52:18.772328 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1c96cdddd3443b8b67f57f4d6c4846e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.755201 883 tablet_service.cc:1468] Processing CreateTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.772984 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f2bef7baa54f1a96b3b59921d59457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.774509 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e181a3f1ae2c49349718d842c166b1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.776201 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34544157495f4e14a7978e0eec70b522. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:18.777626 985 leader_election.cc:343] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 365ef91e7bac43e283602d3d036853ff
W20250625 07:52:18.777865 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 108927us
I20250625 07:52:18.781111 986 leader_election.cc:304] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:18.781481 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 116130us
I20250625 07:52:18.781870 751 tablet_service.cc:1468] Processing CreateTablet for tablet bee1c7f7746a4d6fb763aedadc861421 (DEFAULT_TABLE table=default.Table_3728160a80af42f395da276c742fe44f [id=bbe5def298bc4175997fb660f1bdf381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.782829 748 tablet_service.cc:1468] Processing CreateTablet for tablet 26db0effa17b4cacb1450a01f9631ddf (DEFAULT_TABLE table=default.Table_f00b99b9279946bab7c178245d31c414 [id=5b100b36de77408ebb66b9d7cd64afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.783241 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db0effa17b4cacb1450a01f9631ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.783816 752 tablet_service.cc:1468] Processing CreateTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:18.785578 986 leader_election.cc:343] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 365ef91e7bac43e283602d3d036853ff
I20250625 07:52:18.785741 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f2bef7baa54f1a96b3b59921d59457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.787679 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee1c7f7746a4d6fb763aedadc861421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.788785 1373 raft_consensus.cc:2747] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.789613 1485 leader_election.cc:290] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:18.795589 1163 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.795856 1163 ts_tablet_manager.cc:1397] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.157s user 0.009s sys 0.008s
I20250625 07:52:18.796865 1163 raft_consensus.cc:357] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.797151 1163 raft_consensus.cc:383] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.797294 1163 raft_consensus.cc:738] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.797578 1163 consensus_queue.cc:260] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.798277 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a6c0834adf1748c6a67805d26b973171 (DEFAULT_TABLE table=default.Table_1444a3af50a54dfe9739e5c255dc27a4 [id=8e2d9c1cd8cd4b149d3be543ab086667]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.799296 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:18.799961 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:18.801949 1163 ts_tablet_manager.cc:1428] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:18.807572 1163 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.808203 1160 ts_tablet_manager.cc:1939] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:18.809698 1163 tablet_bootstrap.cc:654] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.818269 1166 ts_tablet_manager.cc:1918] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.819866 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:18.824039 1163 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:18.824604 713 leader_election.cc:343] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:18.820958 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34a338005a7b44b9b0fba7a7096cfcb8 consensus state has no leader
I20250625 07:52:18.820683 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f4184bdee984426ea3612ca34ca1bf92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:18.829811 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_adb5d0acb48c45b781c468e36e59b4c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.830049 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4184bdee984426ea3612ca34ca1bf92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.830302 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adb5d0acb48c45b781c468e36e59b4c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:18.821168 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcc3afc210a540058294121d2b83c66d consensus state has no leader
I20250625 07:52:18.831542 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d7220ef926924a20949121c91fa08801" } new_table_name: "default.Table_91ecb03f4d35473da357098c9dd96216" modify_external_catalogs: true
I20250625 07:52:18.823900 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1c8694f09dcf47ab9f5fc8854c21c331
I20250625 07:52:18.828339 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 26d2bb8b0a8f45079083f2b42bc25011 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:18.833261 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dac905b911e84c5cad6ab113b3907866" } new_table_name: "default.Table_7476f8055aa048e9b8c30b2269f4f2c9" modify_external_catalogs: true
W20250625 07:52:18.833460 713 leader_election.cc:343] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.834838 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.835685 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2560f175e7af433db12e4a7c5b4c3c69" } modify_external_catalogs: true
I20250625 07:52:18.835855 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2560f175e7af433db12e4a7c5b4c3c69" } modify_external_catalogs: true
I20250625 07:52:18.836562 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_df524f8ba5984db8a7f578d01cd2bcd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:18.839294 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df524f8ba5984db8a7f578d01cd2bcd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:18.840430 1163 ts_tablet_manager.cc:1397] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.004s
I20250625 07:52:18.841337 1163 raft_consensus.cc:357] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.841550 1163 raft_consensus.cc:383] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.841624 1163 raft_consensus.cc:738] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.842180 1163 consensus_queue.cc:260] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.847405 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:18.847959 1163 ts_tablet_manager.cc:1428] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 07:52:18.850114 1158 tablet_bootstrap.cc:654] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.850163 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:18.785701 986 leader_election.cc:304] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:18.851126 1163 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.853572 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.855453 1163 tablet_bootstrap.cc:654] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.856040 883 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.861543 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:18.862087 1477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb542ac24f6405687c1067f2885bec6. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:18.862365 1477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:18.862538 1503 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:18.862788 1503 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:18.867856 1503 leader_election.cc:290] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317), 3eb542ac24f6405687c1067f2885bec6 (127.0.104.60:33457)
I20250625 07:52:18.876700 1166 log.cc:1199] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/aec412a15b964060a24631e7ebcf1ef7
I20250625 07:52:18.877251 1166 ts_tablet_manager.cc:1939] T aec412a15b964060a24631e7ebcf1ef7 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:18.880151 1517 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 07:52:18.883085 1163 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.883399 1163 ts_tablet_manager.cc:1397] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
I20250625 07:52:18.884467 1163 raft_consensus.cc:357] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.884804 1163 raft_consensus.cc:383] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.884979 1163 raft_consensus.cc:738] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.885322 1163 consensus_queue.cc:260] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:18.887755 985 leader_election.cc:343] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 86ea9df068a94d9187f2f9620e371a39
W20250625 07:52:18.887938 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 84495.2us
I20250625 07:52:18.889400 1015 tablet_service.cc:1468] Processing CreateTablet for tablet bee1c7f7746a4d6fb763aedadc861421 (DEFAULT_TABLE table=default.Table_3728160a80af42f395da276c742fe44f [id=bbe5def298bc4175997fb660f1bdf381]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.889765 1485 raft_consensus.cc:2747] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.890759 1163 ts_tablet_manager.cc:1428] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 07:52:18.894791 1163 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.897297 1163 tablet_bootstrap.cc:654] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.909072 1413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39212 (request call id 0) took 1456 ms. Trace:
I20250625 07:52:18.913848 1162 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:18.914602 1162 ts_tablet_manager.cc:1397] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.488s user 0.003s sys 0.003s
I20250625 07:52:18.923107 1162 raft_consensus.cc:357] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.926406 1162 raft_consensus.cc:383] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.926789 1163 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.927062 1163 ts_tablet_manager.cc:1397] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.011s
I20250625 07:52:18.927268 1162 raft_consensus.cc:738] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.928304 1163 raft_consensus.cc:357] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.928745 1163 raft_consensus.cc:383] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.928897 1163 raft_consensus.cc:738] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.929332 1163 consensus_queue.cc:260] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.930554 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e181a3f1ae2c49349718d842c166b1e6 (DEFAULT_TABLE table=default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.929800 1162 consensus_queue.cc:260] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:18.932017 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 69153us
W20250625 07:52:18.932291 986 leader_election.cc:343] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b7139335c7124ab79bc2d0a251984199
I20250625 07:52:18.935344 1163 ts_tablet_manager.cc:1428] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250625 07:52:18.935806 1391 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "191b611b772a4fda9d1cba98776138a7" instance_seqno: 1750837930221676) as {username='slave'} at 127.0.104.1:36711; Asking this server to re-register.
I20250625 07:52:18.936813 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 34544157495f4e14a7978e0eec70b522 (DEFAULT_TABLE table=default.Table_d057d5343c334e9e92d5d9d977f90e6a [id=04e37ec20f144d578c85368b35713aa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.937830 824 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:18.939543 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eea1c257b584a029efa08922547b0cc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:18.940407 1391 ts_manager.cc:194] Registered new tserver with Master: 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:18.940605 1163 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:18.940163 713 leader_election.cc:343] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:18.938520 1162 ts_tablet_manager.cc:1428] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.003s sys 0.001s
I20250625 07:52:18.941795 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 482 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:18.942968 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.943240 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:18.943428 1163 tablet_bootstrap.cc:654] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.943746 986 leader_election.cc:304] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:18.945202 1162 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:18.948782 1485 raft_consensus.cc:2747] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.909268 1413 rpcz_store.cc:276] 0625 07:52:17.453079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.453142 (+ 63us) service_pool.cc:225] Handling call
0625 07:52:17.972441 (+519299us) raft_consensus.cc:1469] Updating replica for 388 ops
0625 07:52:18.051249 (+ 78808us) raft_consensus.cc:1532] Early marking committed up to index 94
0625 07:52:18.174405 (+123156us) raft_consensus.cc:1537] Triggering prepare for 388 ops
0625 07:52:18.175130 (+ 725us) write_op.cc:270] Start()
0625 07:52:18.175188 (+ 58us) write_op.cc:276] Timestamp: P: 1750837932341220 usec, L: 0
0625 07:52:18.176635 (+ 1447us) write_op.cc:270] Start()
0625 07:52:18.176682 (+ 47us) write_op.cc:276] Timestamp: P: 1750837932341745 usec, L: 0
0625 07:52:18.178393 (+ 1711us) write_op.cc:270] Start()
0625 07:52:18.178428 (+ 35us) write_op.cc:276] Timestamp: P: 1750837932342361 usec, L: 0
0625 07:52:18.179083 (+ 655us) write_op.cc:270] Start()
0625 07:52:18.179109 (+ 26us) write_op.cc:276] Timestamp: P: 1750837932348522 usec, L: 0
0625 07:52:18.179685 (+ 576us) write_op.cc:270] Start()
0625 07:52:18.179708 (+ 23us) write_op.cc:276] Timestamp: P: 1750837932350087 usec, L: 0
0625 07:52:18.180277 (+ 569us) write_op.cc:270] Start()
0625 07:52:18.180306 (+ 29us) write_op.cc:276] Timestamp: P: 1750837932387138 usec, L: 0
0625 07:52:18.180968 (+ 662us) write_op.cc:270] Start()
0625 07:52:18.181015 (+ 47us) write_op.cc:276] Timestamp: P: 1750837932408886 usec, L: 0
0625 07:52:18.181144 (+ 129us) write_op.cc:270] Start()
0625 07:52:18.181162 (+ 18us) write_op.cc:276] Timestamp: P: 1750837932417393 usec, L: 0
0625 07:52:18.181279 (+ 117us) write_op.cc:270] Start()
0625 07:52:18.181295 (+ 16us) write_op.cc:276] Timestamp: P: 1750837932428503 usec, L: 0
0625 07:52:18.181433 (+ 138us) write_op.cc:270] Start()
0625 07:52:18.181450 (+ 17us) write_op.cc:276] Timestamp: P: 1750837932435824 usec, L: 0
0625 07:52:18.181598 (+ 148us) write_op.cc:270] Start()
0625 07:52:18.181622 (+ 24us) write_op.cc:276] Timestamp: P: 1750837932436433 usec, L: 0
0625 07:52:18.181779 (+ 157us) write_op.cc:270] Start()
0625 07:52:18.181799 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932436952 usec, L: 0
0625 07:52:18.181986 (+ 187us) write_op.cc:270] Start()
0625 07:52:18.182016 (+ 30us) write_op.cc:276] Timestamp: P: 1750837932437473 usec, L: 0
0625 07:52:18.182144 (+ 128us) write_op.cc:270] Start()
0625 07:52:18.182163 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932437939 usec, L: 0
0625 07:52:18.182304 (+ 141us) write_op.cc:270] Start()
0625 07:52:18.182323 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932453344 usec, L: 0
0625 07:52:18.182444 (+ 121us) write_op.cc:270] Start()
0625 07:52:18.182474 (+ 30us) write_op.cc:276] Timestamp: P: 1750837932459397 usec, L: 0
0625 07:52:18.182692 (+ 218us) write_op.cc:270] Start()
0625 07:52:18.182713 (+ 21us) write_op.cc:276] Timestamp: P: 1750837932509701 usec, L: 0
0625 07:52:18.182867 (+ 154us) write_op.cc:270] Start()
0625 07:52:18.182891 (+ 24us) write_op.cc:276] Timestamp: P: 1750837932510711 usec, L: 0
0625 07:52:18.183016 (+ 125us) write_op.cc:270] Start()
0625 07:52:18.183035 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932511174 usec, L: 0
0625 07:52:18.183160 (+ 125us) write_op.cc:270] Start()
0625 07:52:18.183179 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932511701 usec, L: 0
0625 07:52:18.183311 (+ 132us) write_op.cc:270] Start()
0625 07:52:18.183332 (+ 21us) write_op.cc:276] Timestamp: P: 1750837932512237 usec, L: 0
0625 07:52:18.183474 (+ 142us) write_op.cc:270] Start()
0625 07:52:18.183499 (+ 25us) write_op.cc:276] Timestamp: P: 1750837932523224 usec, L: 0
0625 07:52:18.183654 (+ 155us) write_op.cc:270] Start()
0625 07:52:18.183674 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932523742 usec, L: 0
0625 07:52:18.183828 (+ 154us) write_op.cc:270] Start()
0625 07:52:18.183860 (+ 32us) write_op.cc:276] Timestamp: P: 1750837932524271 usec, L: 0
0625 07:52:18.184040 (+ 180us) write_op.cc:270] Start()
0625 07:52:18.184062 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932536860 usec, L: 0
0625 07:52:18.184220 (+ 158us) write_op.cc:270] Start()
0625 07:52:18.184240 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932563244 usec, L: 0
0625 07:52:18.184393 (+ 153us) write_op.cc:270] Start()
0625 07:52:18.184415 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932594182 usec, L: 0
0625 07:52:18.184540 (+ 125us) write_op.cc:270] Start()
0625 07:52:18.184565 (+ 25us) write_op.cc:276] Timestamp: P: 1750837932595822 usec, L: 0
0625 07:52:18.184757 (+ 192us) write_op.cc:270] Start()
0625 07:52:18.184779 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932600468 usec, L: 0
0625 07:52:18.184935 (+ 156us) write_op.cc:270] Start()
0625 07:52:18.184955 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932617918 usec, L: 0
0625 07:52:18.185092 (+ 137us) write_op.cc:270] Start()
0625 07:52:18.185114 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932676934 usec, L: 0
0625 07:52:18.185246 (+ 132us) write_op.cc:270] Start()
0625 07:52:18.185265 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932695946 usec, L: 0
0625 07:52:18.185400 (+ 135us) write_op.cc:270] Start()
0625 07:52:18.185419 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932698539 usec, L: 0
0625 07:52:18.185546 (+ 127us) write_op.cc:270] Start()
0625 07:52:18.185563 (+ 17us) write_op.cc:276] Timestamp: P: 1750837932699151 usec, L: 0
0625 07:52:18.185684 (+ 121us) write_op.cc:270] Start()
0625 07:52:18.185704 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932699653 usec, L: 0
0625 07:52:18.185849 (+ 145us) write_op.cc:270] Start()
0625 07:52:18.185886 (+ 37us) write_op.cc:276] Timestamp: P: 1750837932700118 usec, L: 0
0625 07:52:18.187521 (+ 1635us) write_op.cc:270] Start()
0625 07:52:18.187546 (+ 25us) write_op.cc:276] Timestamp: P: 1750837932700643 usec, L: 0
0625 07:52:18.187727 (+ 181us) write_op.cc:270] Start()
0625 07:52:18.187750 (+ 23us) write_op.cc:276] Timestamp: P: 1750837932718242 usec, L: 0
0625 07:52:18.188486 (+ 736us) write_op.cc:270] Start()
0625 07:52:18.188511 (+ 25us) write_op.cc:276] Timestamp: P: 1750837932718854 usec, L: 0
0625 07:52:18.188670 (+ 159us) write_op.cc:270] Start()
0625 07:52:18.188738 (+ 68us) write_op.cc:276] Timestamp: P: 1750837932719321 usec, L: 0
0625 07:52:18.188900 (+ 162us) write_op.cc:270] Start()
0625 07:52:18.188922 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932719848 usec, L: 0
0625 07:52:18.189058 (+ 136us) write_op.cc:270] Start()
0625 07:52:18.189075 (+ 17us) write_op.cc:276] Timestamp: P: 1750837932724411 usec, L: 0
0625 07:52:18.189288 (+ 213us) write_op.cc:270] Start()
0625 07:52:18.189308 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932744627 usec, L: 0
0625 07:52:18.189439 (+ 131us) write_op.cc:270] Start()
0625 07:52:18.189459 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932745444 usec, L: 0
0625 07:52:18.189589 (+ 130us) write_op.cc:270] Start()
0625 07:52:18.189608 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932745985 usec, L: 0
0625 07:52:18.190170 (+ 562us) write_op.cc:270] Start()
0625 07:52:18.190197 (+ 27us) write_op.cc:276] Timestamp: P: 1750837932746562 usec, L: 0
0625 07:52:18.190368 (+ 171us) write_op.cc:270] Start()
0625 07:52:18.190386 (+ 18us) write_op.cc:276] Timestamp: P: 1750837932757984 usec, L: 0
0625 07:52:18.190494 (+ 108us) write_op.cc:270] Start()
0625 07:52:18.190516 (+ 22us) write_op.cc:276] Timestamp: P: 1750837932759425 usec, L: 0
0625 07:52:18.190628 (+ 112us) write_op.cc:270] Start()
0625 07:52:18.190646 (+ 18us) write_op.cc:276] Timestamp: P: 1750837932759954 usec, L: 0
0625 07:52:18.190855 (+ 209us) write_op.cc:270] Start()
0625 07:52:18.190875 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932829548 usec, L: 0
0625 07:52:18.190998 (+ 123us) write_op.cc:270] Start()
0625 07:52:18.191016 (+ 18us) write_op.cc:276] Timestamp: P: 1750837932845170 usec, L: 0
0625 07:52:18.191188 (+ 172us) write_op.cc:270] Start()
0625 07:52:18.191208 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932848657 usec, L: 0
0625 07:52:18.191346 (+ 138us) write_op.cc:270] Start()
0625 07:52:18.191370 (+ 24us) write_op.cc:276] Timestamp: P: 1750837932849642 usec, L: 0
0625 07:52:18.191497 (+ 127us) write_op.cc:270] Start()
0625 07:52:18.191521 (+ 24us) write_op.cc:276] Timestamp: P: 1750837932850525 usec, L: 0
0625 07:52:18.191665 (+ 144us) write_op.cc:270] Start()
0625 07:52:18.191684 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932928521 usec, L: 0
0625 07:52:18.191833 (+ 149us) write_op.cc:270] Start()
0625 07:52:18.191850 (+ 17us) write_op.cc:276] Timestamp: P: 1750837932929286 usec, L: 0
0625 07:52:18.191967 (+ 117us) write_op.cc:270] Start()
0625 07:52:18.191983 (+ 16us) write_op.cc:276] Timestamp: P: 1750837932929911 usec, L: 0
0625 07:52:18.192141 (+ 158us) write_op.cc:270] Start()
0625 07:52:18.192160 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932966898 usec, L: 0
0625 07:52:18.192317 (+ 157us) write_op.cc:270] Start()
0625 07:52:18.192340 (+ 23us) write_op.cc:276] Timestamp: P: 1750837932968316 usec, L: 0
0625 07:52:18.192526 (+ 186us) write_op.cc:270] Start()
0625 07:52:18.192592 (+ 66us) write_op.cc:276] Timestamp: P: 1750837932969302 usec, L: 0
0625 07:52:18.192769 (+ 177us) write_op.cc:270] Start()
0625 07:52:18.192789 (+ 20us) write_op.cc:276] Timestamp: P: 1750837932970270 usec, L: 0
0625 07:52:18.192909 (+ 120us) write_op.cc:270] Start()
0625 07:52:18.192925 (+ 16us) write_op.cc:276] Timestamp: P: 1750837932973566 usec, L: 0
0625 07:52:18.193070 (+ 145us) write_op.cc:270] Start()
0625 07:52:18.193089 (+ 19us) write_op.cc:276] Timestamp: P: 1750837932974676 usec, L: 0
0625 07:52:18.193229 (+ 140us) write_op.cc:270] Start()
0625 07:52:18.193246 (+ 17us) write_op.cc:276] Timestamp: P: 1750837932987096 usec, L: 0
0625 07:52:18.193385 (+ 139us) write_op.cc:270] Start()
0625 07:52:18.193405 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933012854 usec, L: 0
0625 07:52:18.193521 (+ 116us) write_op.cc:270] Start()
0625 07:52:18.193539 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933013434 usec, L: 0
0625 07:52:18.193648 (+ 109us) write_op.cc:270] Start()
0625 07:52:18.193667 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933013964 usec, L: 0
0625 07:52:18.193868 (+ 201us) write_op.cc:270] Start()
0625 07:52:18.193888 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933014454 usec, L: 0
0625 07:52:18.194032 (+ 144us) write_op.cc:270] Start()
0625 07:52:18.194051 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933018463 usec, L: 0
0625 07:52:18.194187 (+ 136us) write_op.cc:270] Start()
0625 07:52:18.194207 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933023665 usec, L: 0
0625 07:52:18.194383 (+ 176us) write_op.cc:270] Start()
0625 07:52:18.194403 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933026096 usec, L: 0
0625 07:52:18.194551 (+ 148us) write_op.cc:270] Start()
0625 07:52:18.194571 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933108006 usec, L: 0
0625 07:52:18.194691 (+ 120us) write_op.cc:270] Start()
0625 07:52:18.194710 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933108668 usec, L: 0
0625 07:52:18.194870 (+ 160us) write_op.cc:270] Start()
0625 07:52:18.194888 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933109320 usec, L: 0
0625 07:52:18.195002 (+ 114us) write_op.cc:270] Start()
0625 07:52:18.195017 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933129719 usec, L: 0
0625 07:52:18.195119 (+ 102us) write_op.cc:270] Start()
0625 07:52:18.195135 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933149146 usec, L: 0
0625 07:52:18.195280 (+ 145us) write_op.cc:270] Start()
0625 07:52:18.195297 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933150144 usec, L: 0
0625 07:52:18.195407 (+ 110us) write_op.cc:270] Start()
0625 07:52:18.195442 (+ 35us) write_op.cc:276] Timestamp: P: 1750837933150686 usec, L: 0
0625 07:52:18.195572 (+ 130us) write_op.cc:270] Start()
0625 07:52:18.195588 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933177631 usec, L: 0
0625 07:52:18.195694 (+ 106us) write_op.cc:270] Start()
0625 07:52:18.195710 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933178872 usec, L: 0
0625 07:52:18.195843 (+ 133us) write_op.cc:270] Start()
0625 07:52:18.195861 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933183530 usec, L: 0
0625 07:52:18.195969 (+ 108us) write_op.cc:270] Start()
0625 07:52:18.195985 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933186266 usec, L: 0
0625 07:52:18.196085 (+ 100us) write_op.cc:270] Start()
0625 07:52:18.196101 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933199984 usec, L: 0
0625 07:52:18.196194 (+ 93us) write_op.cc:270] Start()
0625 07:52:18.196209 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933201691 usec, L: 0
0625 07:52:18.196327 (+ 118us) write_op.cc:270] Start()
0625 07:52:18.196343 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933212192 usec, L: 0
0625 07:52:18.196502 (+ 159us) write_op.cc:270] Start()
0625 07:52:18.196522 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933213215 usec, L: 0
0625 07:52:18.196683 (+ 161us) write_op.cc:270] Start()
0625 07:52:18.196725 (+ 42us) write_op.cc:276] Timestamp: P: 1750837933235182 usec, L: 0
0625 07:52:18.196853 (+ 128us) write_op.cc:270] Start()
0625 07:52:18.196870 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933251453 usec, L: 0
0625 07:52:18.197021 (+ 151us) write_op.cc:270] Start()
0625 07:52:18.197043 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933260412 usec, L: 0
0625 07:52:18.215350 (+ 18307us) write_op.cc:270] Start()
0625 07:52:18.215393 (+ 43us) write_op.cc:276] Timestamp: P: 1750837933275498 usec, L: 0
0625 07:52:18.215624 (+ 231us) write_op.cc:270] Start()
0625 07:52:18.215645 (+ 21us) write_op.cc:276] Timestamp: P: 1750837933276093 usec, L: 0
0625 07:52:18.215774 (+ 129us) write_op.cc:270] Start()
0625 07:52:18.215794 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933276637 usec, L: 0
0625 07:52:18.215949 (+ 155us) write_op.cc:270] Start()
0625 07:52:18.215968 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933277176 usec, L: 0
0625 07:52:18.218013 (+ 2045us) write_op.cc:270] Start()
0625 07:52:18.218064 (+ 51us) write_op.cc:276] Timestamp: P: 1750837933320934 usec, L: 0
0625 07:52:18.218277 (+ 213us) write_op.cc:270] Start()
0625 07:52:18.218296 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933322009 usec, L: 0
0625 07:52:18.218444 (+ 148us) write_op.cc:270] Start()
0625 07:52:18.218462 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933322608 usec, L: 0
0625 07:52:18.218564 (+ 102us) write_op.cc:270] Start()
0625 07:52:18.218579 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933361450 usec, L: 0
0625 07:52:18.218685 (+ 106us) write_op.cc:270] Start()
0625 07:52:18.218700 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933367998 usec, L: 0
0625 07:52:18.218808 (+ 108us) write_op.cc:270] Start()
0625 07:52:18.218824 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933374317 usec, L: 0
0625 07:52:18.218921 (+ 97us) write_op.cc:270] Start()
0625 07:52:18.218936 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933385469 usec, L: 0
0625 07:52:18.219076 (+ 140us) write_op.cc:270] Start()
0625 07:52:18.219093 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933392989 usec, L: 0
0625 07:52:18.219192 (+ 99us) write_op.cc:270] Start()
0625 07:52:18.219207 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933393740 usec, L: 0
0625 07:52:18.219310 (+ 103us) write_op.cc:270] Start()
0625 07:52:18.219326 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933401243 usec, L: 0
0625 07:52:18.219446 (+ 120us) write_op.cc:270] Start()
0625 07:52:18.219461 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933419302 usec, L: 0
0625 07:52:18.219554 (+ 93us) write_op.cc:270] Start()
0625 07:52:18.219569 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933419945 usec, L: 0
0625 07:52:18.219681 (+ 112us) write_op.cc:270] Start()
0625 07:52:18.219696 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933447096 usec, L: 0
0625 07:52:18.219794 (+ 98us) write_op.cc:270] Start()
0625 07:52:18.219816 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933464134 usec, L: 0
0625 07:52:18.219915 (+ 99us) write_op.cc:270] Start()
0625 07:52:18.219931 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933470055 usec, L: 0
0625 07:52:18.220037 (+ 106us) write_op.cc:270] Start()
0625 07:52:18.220051 (+ 14us) write_op.cc:276] Timestamp: P: 1750837933470940 usec, L: 0
0625 07:52:18.220190 (+ 139us) write_op.cc:270] Start()
0625 07:52:18.220207 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933473297 usec, L: 0
0625 07:52:18.220306 (+ 99us) write_op.cc:270] Start()
0625 07:52:18.220322 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933475049 usec, L: 0
0625 07:52:18.220419 (+ 97us) write_op.cc:270] Start()
0625 07:52:18.220433 (+ 14us) write_op.cc:276] Timestamp: P: 1750837933496021 usec, L: 0
0625 07:52:18.220529 (+ 96us) write_op.cc:270] Start()
0625 07:52:18.220545 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933529615 usec, L: 0
0625 07:52:18.220644 (+ 99us) write_op.cc:270] Start()
0625 07:52:18.220660 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933531683 usec, L: 0
0625 07:52:18.265842 (+ 45182us) write_op.cc:270] Start()
0625 07:52:18.265880 (+ 38us) write_op.cc:276] Timestamp: P: 1750837933532599 usec, L: 0
0625 07:52:18.266126 (+ 246us) write_op.cc:270] Start()
0625 07:52:18.266158 (+ 32us) write_op.cc:276] Timestamp: P: 1750837933536740 usec, L: 0
0625 07:52:18.266327 (+ 169us) write_op.cc:270] Start()
0625 07:52:18.266349 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933543676 usec, L: 0
0625 07:52:18.266487 (+ 138us) write_op.cc:270] Start()
0625 07:52:18.266508 (+ 21us) write_op.cc:276] Timestamp: P: 1750837933580803 usec, L: 0
0625 07:52:18.266658 (+ 150us) write_op.cc:270] Start()
0625 07:52:18.266680 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933595350 usec, L: 0
0625 07:52:18.269392 (+ 2712us) write_op.cc:270] Start()
0625 07:52:18.269419 (+ 27us) write_op.cc:276] Timestamp: P: 1750837933602432 usec, L: 0
0625 07:52:18.269550 (+ 131us) write_op.cc:270] Start()
0625 07:52:18.269570 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933624556 usec, L: 0
0625 07:52:18.269720 (+ 150us) write_op.cc:270] Start()
0625 07:52:18.269749 (+ 29us) write_op.cc:276] Timestamp: P: 1750837933638297 usec, L: 0
0625 07:52:18.269890 (+ 141us) write_op.cc:270] Start()
0625 07:52:18.269908 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933642137 usec, L: 0
0625 07:52:18.270020 (+ 112us) write_op.cc:270] Start()
0625 07:52:18.270038 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933646911 usec, L: 0
0625 07:52:18.270171 (+ 133us) write_op.cc:270] Start()
0625 07:52:18.270189 (+ 18us) write_op.cc:276] Timestamp: P: 1750837933651723 usec, L: 0
0625 07:52:18.270306 (+ 117us) write_op.cc:270] Start()
0625 07:52:18.270323 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933697345 usec, L: 0
0625 07:52:18.270502 (+ 179us) write_op.cc:270] Start()
0625 07:52:18.270524 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933723599 usec, L: 0
0625 07:52:18.270677 (+ 153us) write_op.cc:270] Start()
0625 07:52:18.270699 (+ 22us) write_op.cc:276] Timestamp: P: 1750837933725531 usec, L: 0
0625 07:52:18.270865 (+ 166us) write_op.cc:270] Start()
0625 07:52:18.270885 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933727805 usec, L: 0
0625 07:52:18.270999 (+ 114us) write_op.cc:270] Start()
0625 07:52:18.271016 (+ 17us) write_op.cc:276] Timestamp: P: 1750837933728426 usec, L: 0
0625 07:52:18.271129 (+ 113us) write_op.cc:270] Start()
0625 07:52:18.271178 (+ 49us) write_op.cc:276] Timestamp: P: 1750837933729080 usec, L: 0
0625 07:52:18.271292 (+ 114us) write_op.cc:270] Start()
0625 07:52:18.271308 (+ 16us) write_op.cc:276] Timestamp: P: 1750837933741954 usec, L: 0
0625 07:52:18.271421 (+ 113us) write_op.cc:270] Start()
0625 07:52:18.271436 (+ 15us) write_op.cc:276] Timestamp: P: 1750837933745763 usec, L: 0
0625 07:52:18.271563 (+ 127us) write_op.cc:270] Start()
0625 07:52:18.271586 (+ 23us) write_op.cc:276] Timestamp: P: 1750837933749210 usec, L: 0
0625 07:52:18.271729 (+ 143us) write_op.cc:270] Start()
0625 07:52:18.271749 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933776256 usec, L: 0
0625 07:52:18.271898 (+ 149us) write_op.cc:270] Start()
0625 07:52:18.271919 (+ 21us) write_op.cc:276] Timestamp: P: 1750837933776936 usec, L: 0
0625 07:52:18.272056 (+ 137us) write_op.cc:270] Start()
0625 07:52:18.272077 (+ 21us) write_op.cc:276] Timestamp: P: 1750837933777495 usec, L: 0
0625 07:52:18.272229 (+ 152us) write_op.cc:270] Start()
0625 07:52:18.283681 (+ 11452us) write_op.cc:276] Timestamp: P: 1750837933777990 usec, L: 0
0625 07:52:18.284205 (+ 524us) write_op.cc:270] Start()
0625 07:52:18.284241 (+ 36us) write_op.cc:276] Timestamp: P: 1750837933793284 usec, L: 0
0625 07:52:18.284426 (+ 185us) write_op.cc:270] Start()
0625 07:52:18.284485 (+ 59us) write_op.cc:276] Timestamp: P: 1750837933815913 usec, L: 0
0625 07:52:18.284661 (+ 176us) write_op.cc:270] Start()
0625 07:52:18.284726 (+ 65us) write_op.cc:276] Timestamp: P: 1750837933816467 usec, L: 0
0625 07:52:18.284911 (+ 185us) write_op.cc:270] Start()
0625 07:52:18.284939 (+ 28us) write_op.cc:276] Timestamp: P: 1750837933823411 usec, L: 0
0625 07:52:18.285103 (+ 164us) write_op.cc:270] Start()
0625 07:52:18.285129 (+ 26us) write_op.cc:276] Timestamp: P: 1750837933836015 usec, L: 0
0625 07:52:18.285291 (+ 162us) write_op.cc:270] Start()
0625 07:52:18.285315 (+ 24us) write_op.cc:276] Timestamp: P: 1750837933864992 usec, L: 0
0625 07:52:18.285437 (+ 122us) write_op.cc:270] Start()
0625 07:52:18.285456 (+ 19us) write_op.cc:276] Timestamp: P: 1750837933902668 usec, L: 0
0625 07:52:18.285598 (+ 142us) write_op.cc:270] Start()
0625 07:52:18.285621 (+ 23us) write_op.cc:276] Timestamp: P: 1750837933949880 usec, L: 0
0625 07:52:18.285790 (+ 169us) write_op.cc:270] Start()
0625 07:52:18.285825 (+ 35us) write_op.cc:276] Timestamp: P: 1750837933950606 usec, L: 0
0625 07:52:18.285980 (+ 155us) write_op.cc:270] Start()
0625 07:52:18.286006 (+ 26us) write_op.cc:276] Timestamp: P: 1750837933954317 usec, L: 0
0625 07:52:18.286163 (+ 157us) write_op.cc:270] Start()
0625 07:52:18.286184 (+ 21us) write_op.cc:276] Timestamp: P: 1750837933960992 usec, L: 0
0625 07:52:18.286319 (+ 135us) write_op.cc:270] Start()
0625 07:52:18.286339 (+ 20us) write_op.cc:276] Timestamp: P: 1750837933983251 usec, L: 0
0625 07:52:18.286459 (+ 120us) write_op.cc:270] Start()
0625 07:52:18.286477 (+ 18us) write_op.cc:276] Timestamp: P: 1750837934003379 usec, L: 0
0625 07:52:18.286633 (+ 156us) write_op.cc:270] Start()
0625 07:52:18.286654 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934011802 usec, L: 0
0625 07:52:18.286789 (+ 135us) write_op.cc:270] Start()
0625 07:52:18.286823 (+ 34us) write_op.cc:276] Timestamp: P: 1750837934012511 usec, L: 0
0625 07:52:18.287001 (+ 178us) write_op.cc:270] Start()
0625 07:52:18.287020 (+ 19us) write_op.cc:276] Timestamp: P: 1750837934013112 usec, L: 0
0625 07:52:18.287179 (+ 159us) write_op.cc:270] Start()
0625 07:52:18.287199 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934112571 usec, L: 0
0625 07:52:18.287340 (+ 141us) write_op.cc:270] Start()
0625 07:52:18.287364 (+ 24us) write_op.cc:276] Timestamp: P: 1750837934133208 usec, L: 0
0625 07:52:18.287517 (+ 153us) write_op.cc:270] Start()
0625 07:52:18.287537 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934144225 usec, L: 0
0625 07:52:18.287684 (+ 147us) write_op.cc:270] Start()
0625 07:52:18.287707 (+ 23us) write_op.cc:276] Timestamp: P: 1750837934156593 usec, L: 0
0625 07:52:18.287857 (+ 150us) write_op.cc:270] Start()
0625 07:52:18.287877 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934165250 usec, L: 0
0625 07:52:18.288019 (+ 142us) write_op.cc:270] Start()
0625 07:52:18.288043 (+ 24us) write_op.cc:276] Timestamp: P: 1750837934172778 usec, L: 0
0625 07:52:18.288180 (+ 137us) write_op.cc:270] Start()
0625 07:52:18.288199 (+ 19us) write_op.cc:276] Timestamp: P: 1750837934180294 usec, L: 0
0625 07:52:18.288337 (+ 138us) write_op.cc:270] Start()
0625 07:52:18.288360 (+ 23us) write_op.cc:276] Timestamp: P: 1750837934188439 usec, L: 0
0625 07:52:18.288498 (+ 138us) write_op.cc:270] Start()
0625 07:52:18.288522 (+ 24us) write_op.cc:276] Timestamp: P: 1750837934190173 usec, L: 0
0625 07:52:18.289070 (+ 548us) write_op.cc:270] Start()
0625 07:52:18.289094 (+ 24us) write_op.cc:276] Timestamp: P: 1750837934191103 usec, L: 0
0625 07:52:18.289240 (+ 146us) write_op.cc:270] Start()
0625 07:52:18.289260 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934192515 usec, L: 0
0625 07:52:18.289445 (+ 185us) write_op.cc:270] Start()
0625 07:52:18.289501 (+ 56us) write_op.cc:276] Timestamp: P: 1750837934193382 usec, L: 0
0625 07:52:18.289707 (+ 206us) write_op.cc:270] Start()
0625 07:52:18.289729 (+ 22us) write_op.cc:276] Timestamp: P: 1750837934194191 usec, L: 0
0625 07:52:18.290108 (+ 379us) write_op.cc:270] Start()
0625 07:52:18.290131 (+ 23us) write_op.cc:276] Timestamp: P: 1750837934195272 usec, L: 0
0625 07:52:18.290581 (+ 450us) write_op.cc:270] Start()
0625 07:52:18.290607 (+ 26us) write_op.cc:276] Timestamp: P: 1750837934237353 usec, L: 0
0625 07:52:18.290813 (+ 206us) write_op.cc:270] Start()
0625 07:52:18.290842 (+ 29us) write_op.cc:276] Timestamp: P: 1750837934238035 usec, L: 0
0625 07:52:18.290998 (+ 156us) write_op.cc:270] Start()
0625 07:52:18.291018 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934243573 usec, L: 0
0625 07:52:18.291155 (+ 137us) write_op.cc:270] Start()
0625 07:52:18.291177 (+ 22us) write_op.cc:276] Timestamp: P: 1750837934259846 usec, L: 0
0625 07:52:18.291349 (+ 172us) write_op.cc:270] Start()
0625 07:52:18.291371 (+ 22us) write_op.cc:276] Timestamp: P: 1750837934261195 usec, L: 0
0625 07:52:18.291504 (+ 133us) write_op.cc:270] Start()
0625 07:52:18.291524 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934269324 usec, L: 0
0625 07:52:18.291661 (+ 137us) write_op.cc:270] Start()
0625 07:52:18.291682 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934270150 usec, L: 0
0625 07:52:18.291847 (+ 165us) write_op.cc:270] Start()
0625 07:52:18.291867 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934270902 usec, L: 0
0625 07:52:18.292008 (+ 141us) write_op.cc:270] Start()
0625 07:52:18.292029 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934329325 usec, L: 0
0625 07:52:18.292187 (+ 158us) write_op.cc:270] Start()
0625 07:52:18.292211 (+ 24us) write_op.cc:276] Timestamp: P: 1750837934349441 usec, L: 0
0625 07:52:18.292343 (+ 132us) write_op.cc:270] Start()
0625 07:52:18.292361 (+ 18us) write_op.cc:276] Timestamp: P: 1750837934358780 usec, L: 0
0625 07:52:18.292496 (+ 135us) write_op.cc:270] Start()
0625 07:52:18.292515 (+ 19us) write_op.cc:276] Timestamp: P: 1750837934379409 usec, L: 0
0625 07:52:18.292684 (+ 169us) write_op.cc:270] Start()
0625 07:52:18.292728 (+ 44us) write_op.cc:276] Timestamp: P: 1750837934381847 usec, L: 0
0625 07:52:18.292881 (+ 153us) write_op.cc:270] Start()
0625 07:52:18.292902 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934382413 usec, L: 0
0625 07:52:18.293046 (+ 144us) write_op.cc:270] Start()
0625 07:52:18.293066 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934385114 usec, L: 0
0625 07:52:18.293198 (+ 132us) write_op.cc:270] Start()
0625 07:52:18.293216 (+ 18us) write_op.cc:276] Timestamp: P: 1750837934405739 usec, L: 0
0625 07:52:18.293337 (+ 121us) write_op.cc:270] Start()
0625 07:52:18.293357 (+ 20us) write_op.cc:276] Timestamp: P: 1750837934553967 usec, L: 0
0625 07:52:18.293502 (+ 145us) write_op.cc:270] Start()
0625 07:52:18.293519 (+ 17us) write_op.cc:276] Timestamp: P: 1750837934555096 usec, L: 0
0625 07:52:18.293632 (+ 113us) write_op.cc:270] Start()
0625 07:52:18.293651 (+ 19us) write_op.cc:276] Timestamp: P: 1750837934556336 usec, L: 0
0625 07:52:18.297747 (+ 4096us) write_op.cc:270] Start()
0625 07:52:18.297774 (+ 27us) write_op.cc:276] Timestamp: P: 1750837934557259 usec, L: 0
0625 07:52:18.298130 (+ 356us) write_op.cc:270] Start()
0625 07:52:18.298155 (+ 25us) write_op.cc:276] Timestamp: P: 1750837934564408 usec, L: 0
0625 07:52:18.298396 (+ 241us) write_op.cc:270] Start()
0625 07:52:18.298418 (+ 22us) write_op.cc:276] Timestamp: P: 1750837934565004 usec, L: 0
0625 07:52:18.298560 (+ 142us) write_op.cc:270] Start()
0625 07:52:18.298581 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934565521 usec, L: 0
0625 07:52:18.298723 (+ 142us) write_op.cc:270] Start()
0625 07:52:18.298744 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934592166 usec, L: 0
0625 07:52:18.298887 (+ 143us) write_op.cc:270] Start()
0625 07:52:18.298905 (+ 18us) write_op.cc:276] Timestamp: P: 1750837934597498 usec, L: 0
0625 07:52:18.299014 (+ 109us) write_op.cc:270] Start()
0625 07:52:18.299033 (+ 19us) write_op.cc:276] Timestamp: P: 1750837934610009 usec, L: 0
0625 07:52:18.300821 (+ 1788us) write_op.cc:270] Start()
0625 07:52:18.300850 (+ 29us) write_op.cc:276] Timestamp: P: 1750837934613927 usec, L: 0
0625 07:52:18.301009 (+ 159us) write_op.cc:270] Start()
0625 07:52:18.301030 (+ 21us) write_op.cc:276] Timestamp: P: 1750837934614467 usec, L: 0
0625 07:52:18.301200 (+ 170us) write_op.cc:270] Start()
0625 07:52:18.301228 (+ 28us) write_op.cc:276] Timestamp: P: 1750837
W20250625 07:52:18.956842 986 leader_election.cc:343] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 86ea9df068a94d9187f2f9620e371a39
I20250625 07:52:18.961017 1162 tablet_bootstrap.cc:654] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.963135 1163 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:18.963410 1163 ts_tablet_manager.cc:1397] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20250625 07:52:18.964900 1163 raft_consensus.cc:357] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.965196 1163 raft_consensus.cc:383] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:18.965320 1163 raft_consensus.cc:738] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:18.965623 1163 consensus_queue.cc:260] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:18.968470 1163 ts_tablet_manager.cc:1428] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:18.971339 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 26db0effa17b4cacb1450a01f9631ddf (DEFAULT_TABLE table=default.Table_f00b99b9279946bab7c178245d31c414 [id=5b100b36de77408ebb66b9d7cd64afb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.973044 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 (DEFAULT_TABLE table=default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:18.973409 1163 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:18.974110 986 leader_election.cc:304] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:18.974798 1485 raft_consensus.cc:2747] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:18.976843 1163 tablet_bootstrap.cc:654] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:18.994127 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6c0834adf1748c6a67805d26b973171. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:18.997798 1161 ts_tablet_manager.cc:1918] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:18.998063 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90f2bef7baa54f1a96b3b59921d59457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.001683 1308 raft_consensus.cc:491] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.001955 1308 raft_consensus.cc:513] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.002852 1308 leader_election.cc:290] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.003098 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e181a3f1ae2c49349718d842c166b1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.003784 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d9d66546d7c42949dfe5db77f1057fe" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:19.004248 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d9d66546d7c42949dfe5db77f1057fe" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:19.004778 850 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.004998 851 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.005146 851 leader_election.cc:304] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.005570 1308 raft_consensus.cc:2747] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.008857 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee1c7f7746a4d6fb763aedadc861421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.012835 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34544157495f4e14a7978e0eec70b522. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.016820 1161 log.cc:1199] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/012df7beedb1446ca5e533d6624afc10
I20250625 07:52:19.022969 1308 raft_consensus.cc:491] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.023181 1308 raft_consensus.cc:513] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.023872 1162 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.023939 1308 leader_election.cc:290] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.024056 1162 ts_tablet_manager.cc:1397] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.000s
I20250625 07:52:19.024931 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f63fdf2c8464c8fb58f412263d4e5c5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:19.024963 1162 raft_consensus.cc:357] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.025228 1162 raft_consensus.cc:383] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.025346 1162 raft_consensus.cc:738] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:19.025565 851 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.025647 1162 consensus_queue.cc:260] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.026249 1162 ts_tablet_manager.cc:1428] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:19.026618 1162 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.035243 1162 tablet_bootstrap.cc:654] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.036569 1163 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.037770 1163 ts_tablet_manager.cc:1397] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.022s
I20250625 07:52:19.038924 1163 raft_consensus.cc:357] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.039307 1163 raft_consensus.cc:383] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.039741 1163 raft_consensus.cc:738] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.040158 1163 consensus_queue.cc:260] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:19.043095 1486 log.cc:927] Time spent T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20250625 07:52:19.043411 1163 ts_tablet_manager.cc:1428] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:19.043942 1163 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.046823 1163 tablet_bootstrap.cc:654] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.046772 1466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:19.047027 1466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:19.050017 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1" } modify_external_catalogs: true
I20250625 07:52:19.050710 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1" } modify_external_catalogs: true
I20250625 07:52:19.051604 1162 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.051879 1162 ts_tablet_manager.cc:1397] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20250625 07:52:19.052050 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f63fdf2c8464c8fb58f412263d4e5c5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:19.053728 850 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.053867 850 leader_election.cc:304] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.054287 1308 raft_consensus.cc:2747] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.056768 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db0effa17b4cacb1450a01f9631ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.065635 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (table default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]) successfully deleted
I20250625 07:52:19.066243 1163 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.066486 1163 ts_tablet_manager.cc:1397] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.002s
I20250625 07:52:19.067448 1163 raft_consensus.cc:357] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.067750 1163 raft_consensus.cc:383] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:19.067691 584 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:43317, user_credentials={real_user=slave}} blocked reactor thread for 256660us
I20250625 07:52:19.067848 1163 raft_consensus.cc:738] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.068384 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.068171 1163 consensus_queue.cc:260] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.069145 1163 ts_tablet_manager.cc:1428] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:19.069480 1163 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.072942 1163 tablet_bootstrap.cc:654] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.077630 1162 raft_consensus.cc:357] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.077952 1162 raft_consensus.cc:383] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.078068 1162 raft_consensus.cc:738] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.078363 1162 consensus_queue.cc:260] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.079087 1162 ts_tablet_manager.cc:1428] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20250625 07:52:19.080186 1163 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.080262 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.080449 1163 ts_tablet_manager.cc:1397] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250625 07:52:19.081382 1163 raft_consensus.cc:357] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.081666 1163 raft_consensus.cc:383] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.081794 1163 raft_consensus.cc:738] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.082268 1163 consensus_queue.cc:260] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.082731 1160 tablet_replica.cc:331] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.083158 1162 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.088295 1162 tablet_bootstrap.cc:654] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.090003 1161 ts_tablet_manager.cc:1939] T 012df7beedb1446ca5e533d6624afc10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.090471 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 388) took 1358 ms. Trace:
I20250625 07:52:19.083150 1163 ts_tablet_manager.cc:1428] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:19.092355 1163 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.094806 1163 tablet_bootstrap.cc:654] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.095981 1161 rpcz_store.cc:276] 0625 07:52:17.731928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.732235 (+ 307us) service_pool.cc:225] Handling call
0625 07:52:18.410759 (+678524us) ts_tablet_manager.cc:1145] Deleting tablet 012df7beedb1446ca5e533d6624afc10
0625 07:52:18.410766 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:18.968229 (+557463us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:18.997642 (+ 29413us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.090425 (+ 92783us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3522,"tablet-delete.queue_time_us":678269}
I20250625 07:52:19.096972 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 012df7beedb1446ca5e533d6624afc10 (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:19.097533 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 389) took 1347 ms. Trace:
I20250625 07:52:19.106881 1160 raft_consensus.cc:2241] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.107564 1160 raft_consensus.cc:2270] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.108767 1160 ts_tablet_manager.cc:1905] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.111009 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet aec412a15b964060a24631e7ebcf1ef7 (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:19.112851 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 482 } } ignore_live_leader: false dest_uuid: "3eb542ac24f6405687c1067f2885bec6" is_pre_election: true
W20250625 07:52:19.113803 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9a90bb52358f4c38a540ad025e9e0430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a90bb52358f4c38a540ad025e9e0430 already in progress: creating tablet
I20250625 07:52:19.114131 1370 leader_election.cc:304] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a; no voters: 3eb542ac24f6405687c1067f2885bec6, 789c06e7de2842a1b40e97e0778ab6c9
W20250625 07:52:19.119567 713 consensus_peers.cc:487] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:19.102249 1161 rpcz_store.cc:276] 0625 07:52:17.749582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.749638 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:19.097357 (+1347719us) ts_tablet_manager.cc:1145] Deleting tablet 743123ffcf8240dc82bea7178559b7ea
0625 07:52:19.097368 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.097492 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347490}
I20250625 07:52:19.117151 1308 raft_consensus.cc:491] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.119174 914 raft_consensus.cc:1273] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:19.126415 1308 raft_consensus.cc:513] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.127218 1308 leader_election.cc:290] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.127166 1506 consensus_queue.cc:1035] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:19.127828 1161 tablet_replica.cc:331] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.128006 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b377c8111c9f4ac2ba05e65694d611b9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:19.128342 1161 raft_consensus.cc:2241] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.128399 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b377c8111c9f4ac2ba05e65694d611b9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.128557 1161 raft_consensus.cc:2270] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:19.128567 851 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.129282 850 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.129426 850 leader_election.cc:304] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.129957 1308 raft_consensus.cc:2747] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:19.122363 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 743123ffcf8240dc82bea7178559b7ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 743123ffcf8240dc82bea7178559b7ea already in progress: creating tablet
W20250625 07:52:19.134140 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 547b758c2e6c4558bb232f14b37aa81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 547b758c2e6c4558bb232f14b37aa81a already in progress: creating tablet
W20250625 07:52:19.134446 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e181a3f1ae2c49349718d842c166b1e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e181a3f1ae2c49349718d842c166b1e6 already in progress: creating tablet
W20250625 07:52:19.134738 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1c8694f09dcf47ab9f5fc8854c21c331 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c8694f09dcf47ab9f5fc8854c21c331 already in progress: creating tablet
W20250625 07:52:19.134974 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9f1d251171314118b66e3a921a4a50c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f1d251171314118b66e3a921a4a50c5 already in progress: creating tablet
I20250625 07:52:19.137128 1162 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.137367 1162 ts_tablet_manager.cc:1397] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.002s
I20250625 07:52:19.138813 1162 raft_consensus.cc:357] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.142529 1162 raft_consensus.cc:383] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.143124 1162 raft_consensus.cc:738] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.143772 1162 consensus_queue.cc:260] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.149041 1162 ts_tablet_manager.cc:1428] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 07:52:19.150626 1506 raft_consensus.cc:491] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.151146 1506 raft_consensus.cc:513] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.151695 1162 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.154222 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b769ac57346a427abd22214eebc0d324" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.154520 1162 tablet_bootstrap.cc:654] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:19.154985 713 leader_election.cc:343] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.151919 1506 leader_election.cc:290] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.159022 1158 tablet_bootstrap.cc:492] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:19.159438 1308 raft_consensus.cc:491] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.159678 1308 raft_consensus.cc:513] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.160306 1308 leader_election.cc:290] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:19.160929 713 leader_election.cc:343] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b769ac57346a427abd22214eebc0d324
I20250625 07:52:19.161098 713 leader_election.cc:304] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.161756 1506 raft_consensus.cc:2747] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.162060 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de9ccb0a2d3846228acbc2769786a059" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.162776 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de9ccb0a2d3846228acbc2769786a059" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.163484 851 leader_election.cc:343] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.163908 850 leader_election.cc:343] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.164034 850 leader_election.cc:304] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.164330 1160 ts_tablet_manager.cc:1918] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.164455 1160 log.cc:1199] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1e6c2988baa14cbc8c878f79e942880d
I20250625 07:52:19.164599 1308 raft_consensus.cc:2747] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.164902 1160 ts_tablet_manager.cc:1939] T 1e6c2988baa14cbc8c878f79e942880d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.165822 1160 tablet_replica.cc:331] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.166044 1160 raft_consensus.cc:2241] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.166007 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1e6c2988baa14cbc8c878f79e942880d (table default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]) successfully deleted
I20250625 07:52:19.166191 1160 raft_consensus.cc:2270] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.166793 1160 ts_tablet_manager.cc:1905] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.169059 1161 ts_tablet_manager.cc:1905] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.169528 1158 ts_tablet_manager.cc:1397] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.747s user 0.004s sys 0.018s
I20250625 07:52:19.177976 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:19.178766 745 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.179574 888 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:19.180306 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a1baf94a051e485eaf4c0e074e147252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1baf94a051e485eaf4c0e074e147252 already in progress: creating tablet
I20250625 07:52:19.182366 1393 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" instance_seqno: 1750837931198880) as {username='slave'} at 127.0.104.3:49457; Asking this server to re-register.
I20250625 07:52:19.183442 1160 ts_tablet_manager.cc:1918] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.183598 1160 log.cc:1199] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2f63fdf2c8464c8fb58f412263d4e5c5
I20250625 07:52:19.183965 1160 ts_tablet_manager.cc:1939] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.184161 1503 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 07:52:19.185024 888 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.180657 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f3cd40e7670e492bba639148deda5ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.186524 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3cd40e7670e492bba639148deda5ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.180989 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a41ffd566593499299aba5513ccb2f30 consensus state has no leader
I20250625 07:52:19.186960 1392 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "429026e8d98d46339db143f84897a092" instance_seqno: 1750837930718830) as {username='slave'} at 127.0.104.2:43013; Asking this server to re-register.
I20250625 07:52:19.187218 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c65ada52ed03480196c90fc1cd9f3bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.187739 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c65ada52ed03480196c90fc1cd9f3bbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.188095 961 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:19.185459 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (table default.Table_5946e2f2580849578b2bda787459faef [id=e441385dc1044575b17f64b06f14c648]) successfully deleted
W20250625 07:52:19.190140 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9a90bb52358f4c38a540ad025e9e0430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a90bb52358f4c38a540ad025e9e0430 already in progress: creating tablet
I20250625 07:52:19.190431 1308 raft_consensus.cc:491] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.185572 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a8813cc630ad48ca87a8bcc192010644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.190667 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e181a3f1ae2c49349718d842c166b1e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e181a3f1ae2c49349718d842c166b1e6 already in progress: creating tablet
I20250625 07:52:19.190610 1308 raft_consensus.cc:513] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:19.190884 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8813cc630ad48ca87a8bcc192010644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.191306 1308 leader_election.cc:290] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:19.191565 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 547b758c2e6c4558bb232f14b37aa81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 547b758c2e6c4558bb232f14b37aa81a already in progress: creating tablet
I20250625 07:52:19.192142 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.192358 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1c8694f09dcf47ab9f5fc8854c21c331 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c8694f09dcf47ab9f5fc8854c21c331 already in progress: creating tablet
W20250625 07:52:19.192741 851 leader_election.cc:343] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.192893 1160 tablet_replica.cc:331] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.193087 1160 raft_consensus.cc:2241] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.193735 1160 raft_consensus.cc:2270] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.194890 1160 ts_tablet_manager.cc:1905] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.208789 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4de62938b87249fdb2950c71864eeb4c" } new_table_name: "default.Table_80cbf681213f4716b1b7fd920d71b59c" modify_external_catalogs: true
W20250625 07:52:19.210284 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0657163b9a0d4f7f99f57f7fd1f32462 consensus state has no leader
I20250625 07:52:19.210755 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_29b24dec640147299371e6748d35cc5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.211364 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29b24dec640147299371e6748d35cc5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.213212 1160 ts_tablet_manager.cc:1918] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.213171 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1386b0806db34e7482d4483c86eed121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.213761 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1386b0806db34e7482d4483c86eed121 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.214473 1160 log.cc:1199] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:19.215605 1160 ts_tablet_manager.cc:1939] T 7d9d66546d7c42949dfe5db77f1057fe P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.217078 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7d9d66546d7c42949dfe5db77f1057fe (table default.Table_3b26428e16b14c4aaaf95f0f92b3670c [id=60fd5cf37619424381543d3cdc94167d]) successfully deleted
I20250625 07:52:19.219326 1160 tablet_replica.cc:331] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.219830 1160 raft_consensus.cc:2241] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.220438 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f9a2ecb876ab4b089993854b257c3450" } new_table_name: "default.Table_4006274930664fdca4ebf55f2375b00d" modify_external_catalogs: true
I20250625 07:52:19.220921 1160 raft_consensus.cc:2270] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.221380 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0b9db8ced3e74eecb4fc928446f87bc1" } new_table_name: "default.Table_98c9cffe906a4a99bedb93374dab75e4" modify_external_catalogs: true
I20250625 07:52:19.221087 1308 raft_consensus.cc:491] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.221827 1308 raft_consensus.cc:513] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.222738 1308 leader_election.cc:290] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.223871 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81976db0a2f94128a89842a12f067231" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:19.224090 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d375845276c6425baf08c8e94b318afc" } modify_external_catalogs: true
I20250625 07:52:19.224232 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d375845276c6425baf08c8e94b318afc" } modify_external_catalogs: true
W20250625 07:52:19.224649 851 leader_election.cc:343] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.232720 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ccddd630a82341cba7a857e67a349fe0 consensus state has no leader
I20250625 07:52:19.233176 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_77bd933ec7f6471886c2b3454654fe9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.234331 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77bd933ec7f6471886c2b3454654fe9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.238976 1160 ts_tablet_manager.cc:1905] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.239941 1506 raft_consensus.cc:491] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.240132 1506 raft_consensus.cc:513] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.240842 1506 leader_election.cc:290] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:19.242988 713 leader_election.cc:343] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3c8c317c10e14332a3b3919808e4a512
I20250625 07:52:19.246939 1160 ts_tablet_manager.cc:1918] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.247133 1160 log.cc:1199] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b6382963863548398bed90f03b840806
I20250625 07:52:19.247547 1160 ts_tablet_manager.cc:1939] T b6382963863548398bed90f03b840806 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.248124 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b6382963863548398bed90f03b840806 (table default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]) successfully deleted
W20250625 07:52:19.248867 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9f1d251171314118b66e3a921a4a50c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f1d251171314118b66e3a921a4a50c5 already in progress: creating tablet
W20250625 07:52:19.249447 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a1baf94a051e485eaf4c0e074e147252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1baf94a051e485eaf4c0e074e147252 already in progress: creating tablet
I20250625 07:52:19.249763 1160 tablet_replica.cc:331] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.249984 1160 raft_consensus.cc:2241] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.250201 1160 raft_consensus.cc:2270] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.250973 1160 ts_tablet_manager.cc:1905] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.258342 1160 ts_tablet_manager.cc:1918] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.258525 1160 log.cc:1199] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b377c8111c9f4ac2ba05e65694d611b9
I20250625 07:52:19.258989 1160 ts_tablet_manager.cc:1939] T b377c8111c9f4ac2ba05e65694d611b9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.259143 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5fbde73edcc941ebb33753389aea2dc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.259760 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fbde73edcc941ebb33753389aea2dc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.260267 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b377c8111c9f4ac2ba05e65694d611b9 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]) successfully deleted
I20250625 07:52:19.263161 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0cfa01917c304be98bc1f8d5fb704b47" } modify_external_catalogs: true
I20250625 07:52:19.263341 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0cfa01917c304be98bc1f8d5fb704b47" } modify_external_catalogs: true
I20250625 07:52:19.264961 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:19.265763 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5fe3ea5745743019d23d49ff808070a consensus state has no leader
I20250625 07:52:19.266037 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 7d9d66546d7c42949dfe5db77f1057fe from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:19.267055 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f7240577a874a59b3268ff7f0c1beec has no consensus state
I20250625 07:52:19.270218 596 catalog_manager.cc:5582] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:19.270656 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 5de1492e483e47919ee5581ecee227be (table default.Table_3770979326894b6890fc810005775fc1 [id=23322aa9321140c39272b5b2a102fb02]). Expected version 1 got 0
I20250625 07:52:19.270913 596 catalog_manager.cc:5582] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:19.271250 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:19.272243 745 tablet_service.cc:1468] Processing CreateTablet for tablet 62aaff3c000e4e69b52222aa47260c3a (DEFAULT_TABLE table=default.Table_3e66f393d52d48938a989a0c7104879c [id=79d68081c3a643109237f5d5478df82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.273159 888 tablet_service.cc:1468] Processing CreateTablet for tablet 62aaff3c000e4e69b52222aa47260c3a (DEFAULT_TABLE table=default.Table_3e66f393d52d48938a989a0c7104879c [id=79d68081c3a643109237f5d5478df82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.273391 746 tablet_service.cc:1468] Processing CreateTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.273618 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aaff3c000e4e69b52222aa47260c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.274235 744 tablet_service.cc:1468] Processing CreateTablet for tablet 22411725aa5741c497328ce047c44d50 (DEFAULT_TABLE table=default.Table_41047b79cb784bb19f458dbce4573ef5 [id=772213882cb546a8bc37c0b8217eaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.274668 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aaff3c000e4e69b52222aa47260c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.277753 893 tablet_service.cc:1468] Processing CreateTablet for tablet 22411725aa5741c497328ce047c44d50 (DEFAULT_TABLE table=default.Table_41047b79cb784bb19f458dbce4573ef5 [id=772213882cb546a8bc37c0b8217eaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.278086 883 tablet_service.cc:1468] Processing CreateTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.278455 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22411725aa5741c497328ce047c44d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.279174 886 tablet_service.cc:1468] Processing CreateTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 (DEFAULT_TABLE table=default.Table_3e26b4b26dfe434ba65119e99f41bb8c [id=1c84ef7a8f854a9898e3d11ad9d5c675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.279721 894 tablet_service.cc:1468] Processing CreateTablet for tablet 17be34eba92246df973f3d464e13fd30 (DEFAULT_TABLE table=default.Table_6c9346951d87447e84acea414f3cc804 [id=9e4563f37d0a4423887337f954c65aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.280289 895 tablet_service.cc:1468] Processing CreateTablet for tablet d0a75906385744e5ba6938f451a2d7ec (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.281252 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b04a575e0984f879f0179c8baa3f3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.281279 1163 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.281430 1163 ts_tablet_manager.cc:1397] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.189s user 0.004s sys 0.029s
I20250625 07:52:19.282274 1163 raft_consensus.cc:357] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.282457 1163 raft_consensus.cc:383] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.282533 1163 raft_consensus.cc:738] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.282742 1163 consensus_queue.cc:260] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.283245 1163 ts_tablet_manager.cc:1428] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:19.283532 1163 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.286152 752 tablet_service.cc:1468] Processing CreateTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 (DEFAULT_TABLE table=default.Table_3e26b4b26dfe434ba65119e99f41bb8c [id=1c84ef7a8f854a9898e3d11ad9d5c675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.286651 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a49fab6134d059c64f5a3f2b29e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.278173 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22411725aa5741c497328ce047c44d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.289418 751 tablet_service.cc:1468] Processing CreateTablet for tablet 17be34eba92246df973f3d464e13fd30 (DEFAULT_TABLE table=default.Table_6c9346951d87447e84acea414f3cc804 [id=9e4563f37d0a4423887337f954c65aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.289847 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17be34eba92246df973f3d464e13fd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.292857 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17be34eba92246df973f3d464e13fd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.297004 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a49fab6134d059c64f5a3f2b29e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.300956 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a75906385744e5ba6938f451a2d7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.305589 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b04a575e0984f879f0179c8baa3f3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.310087 1392 ts_manager.cc:194] Registered new tserver with Master: 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:19.310912 884 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7240577a874a59b3268ff7f0c1beec (DEFAULT_TABLE table=default.Table_98c9cffe906a4a99bedb93374dab75e4 [id=f453bd82f85947dcb5d1e309ca1b902f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.311317 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7240577a874a59b3268ff7f0c1beec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.317122 757 tablet_service.cc:1468] Processing CreateTablet for tablet d0a75906385744e5ba6938f451a2d7ec (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.317411 754 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7240577a874a59b3268ff7f0c1beec (DEFAULT_TABLE table=default.Table_98c9cffe906a4a99bedb93374dab75e4 [id=f453bd82f85947dcb5d1e309ca1b902f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.318537 1539 mvcc.cc:204] Tried to move back new op lower bound from 7171432198617309184 to 7171432195927900160. Current Snapshot: MvccSnapshot[applied={T|T < 7171432198617309184}]
I20250625 07:52:19.321585 1163 tablet_bootstrap.cc:654] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.332080 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7240577a874a59b3268ff7f0c1beec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.334066 1506 raft_consensus.cc:491] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.337239 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 32a9f4cf6838411fab06fa6d35ae9492
I20250625 07:52:19.337487 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a75906385744e5ba6938f451a2d7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.338217 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 59df0f66116e4dfd9386d107cab0aaf6
I20250625 07:52:19.339496 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce
I20250625 07:52:19.339946 589 catalog_manager.cc:6652] ReplaceTablet: tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce deleted and replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61
I20250625 07:52:19.340370 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 871b603cb92249c6b0f16c825bf46232
I20250625 07:52:19.340660 1506 raft_consensus.cc:513] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.341107 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ea62dfb8f20f4caeaa79ea1e85d99930
I20250625 07:52:19.341893 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.342700 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:19.342947 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.343186 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.343242 590 catalog_manager.cc:6652] ReplaceTablet: tablet 26d2bb8b0a8f45079083f2b42bc25011 deleted and replaced by tablet bfb421ea711a461890e267153250acea
W20250625 07:52:19.345467 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 59df0f66116e4dfd9386d107cab0aaf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59df0f66116e4dfd9386d107cab0aaf6 already in progress: creating tablet
I20250625 07:52:19.348321 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.348366 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:19.349159 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce already in progress: creating tablet
I20250625 07:52:19.354666 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.357172 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81976db0a2f94128a89842a12f067231" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.357182 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.360828 1485 raft_consensus.cc:491] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.361001 1485 raft_consensus.cc:513] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:19.365053 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.368999 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.370980 1096 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:19.371217 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.371472 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c8c317c10e14332a3b3919808e4a512" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.372352 749 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.372530 1485 leader_election.cc:290] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:19.372587 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.373597 1166 tablet_replica.cc:331] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:19.373881 1166 raft_consensus.cc:2241] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.373831 1158 raft_consensus.cc:357] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.374029 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 62aaff3c000e4e69b52222aa47260c3a (DEFAULT_TABLE table=default.Table_3e66f393d52d48938a989a0c7104879c [id=79d68081c3a643109237f5d5478df82c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.374095 1506 leader_election.cc:290] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:19.375160 1166 raft_consensus.cc:2270] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.375224 1160 tablet_replica.cc:331] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.376451 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.376758 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.377157 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 22411725aa5741c497328ce047c44d50 (DEFAULT_TABLE table=default.Table_41047b79cb784bb19f458dbce4573ef5 [id=772213882cb546a8bc37c0b8217eaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.377426 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 (DEFAULT_TABLE table=default.Table_3e26b4b26dfe434ba65119e99f41bb8c [id=1c84ef7a8f854a9898e3d11ad9d5c675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.378014 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 17be34eba92246df973f3d464e13fd30 (DEFAULT_TABLE table=default.Table_6c9346951d87447e84acea414f3cc804 [id=9e4563f37d0a4423887337f954c65aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.378222 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d0a75906385744e5ba6938f451a2d7ec (DEFAULT_TABLE table=default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.378487 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7240577a874a59b3268ff7f0c1beec (DEFAULT_TABLE table=default.Table_98c9cffe906a4a99bedb93374dab75e4 [id=f453bd82f85947dcb5d1e309ca1b902f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.378969 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.379271 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.379526 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.379727 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.379776 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.379904 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aaff3c000e4e69b52222aa47260c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.379907 1166 ts_tablet_manager.cc:1905] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.380147 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.380183 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4c6dba196ca04a6ba70a9180b32eeec3" } modify_external_catalogs: true
I20250625 07:52:19.380342 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.380398 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4c6dba196ca04a6ba70a9180b32eeec3" } modify_external_catalogs: true
I20250625 07:52:19.384830 1158 raft_consensus.cc:383] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.384967 1158 raft_consensus.cc:738] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.385603 1158 consensus_queue.cc:260] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.386270 1158 ts_tablet_manager.cc:1428] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.213s user 0.001s sys 0.001s
I20250625 07:52:19.387715 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a976166b59b44d52a920b760df58965c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.388782 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6c9346951d87447e84acea414f3cc804" } new_table_name: "default.Table_6e6e1b57ae284b1d9bc03891dc267c9c" modify_external_catalogs: true
I20250625 07:52:19.390625 1158 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:19.392381 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7240577a874a59b3268ff7f0c1beec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.392841 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17be34eba92246df973f3d464e13fd30. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:19.399410 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17be34eba92246df973f3d464e13fd30 consensus state has no leader
I20250625 07:52:19.409569 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a75906385744e5ba6938f451a2d7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.410710 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3221b04d095841d4a01f75c2a8004dce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.411731 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e46845b5de944ab384d68468026acafc" } new_table_name: "default.Table_abbbaaa7e3f6474bbb0003c420461b34" modify_external_catalogs: true
I20250625 07:52:19.413709 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_06acb0609d5e417886a27bc6693624ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.415305 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22411725aa5741c497328ce047c44d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.416005 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b04a575e0984f879f0179c8baa3f3fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.416532 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a49fab6134d059c64f5a3f2b29e03. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:19.418378 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e41e53efb6f342d2b1e4d642f7aaad10 consensus state has no leader
W20250625 07:52:19.418881 713 leader_election.cc:343] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.419021 713 leader_election.cc:304] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.419186 1392 ts_manager.cc:194] Registered new tserver with Master: 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:19.419214 713 raft_consensus.cc:2684] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:19.420120 985 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9e77df5e7b3447fdbdb1157e74cf8b6d
I20250625 07:52:19.420477 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0452e73865964f709310fd22ac39a5f0" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.421842 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5cbcfdbba5a84de1ac582e4687f347a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.422425 713 leader_election.cc:343] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.423009 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.423875 1161 ts_tablet_manager.cc:1918] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.424292 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b71aadf074a4a39943b2ff930909f93 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.424481 1162 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.424636 1162 ts_tablet_manager.cc:1397] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.273s user 0.020s sys 0.015s
I20250625 07:52:19.425406 1161 log.cc:1199] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b7139335c7124ab79bc2d0a251984199
I20250625 07:52:19.425498 1162 raft_consensus.cc:357] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.425706 1162 raft_consensus.cc:383] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.425808 1162 raft_consensus.cc:738] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.426087 1162 consensus_queue.cc:260] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.426661 1162 ts_tablet_manager.cc:1428] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:19.426569 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_feabe5a9dd014d698a57f45b19bad2cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.426980 1162 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.433835 1162 tablet_bootstrap.cc:654] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.438704 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3770979326894b6890fc810005775fc1" } new_table_name: "default.Table_0ef24d5b4b124360b654d6903b994151" modify_external_catalogs: true
I20250625 07:52:19.450330 1161 ts_tablet_manager.cc:1939] T b7139335c7124ab79bc2d0a251984199 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.458004 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 390) took 1653 ms. Trace:
I20250625 07:52:19.458328 1161 rpcz_store.cc:276] 0625 07:52:17.797675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.797732 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:19.122059 (+1324327us) ts_tablet_manager.cc:1145] Deleting tablet b7139335c7124ab79bc2d0a251984199
0625 07:52:19.122070 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.283382 (+161312us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.413351 (+129969us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.451153 (+ 37802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1324089}
I20250625 07:52:19.464963 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b7139335c7124ab79bc2d0a251984199 (table default.Table_b2de3b5567ef4b3e9166b04dbd6ef7c0 [id=84244d54b35c47829f8f8a92c3d47ee3]) successfully deleted
I20250625 07:52:19.466114 1161 tablet_replica.cc:331] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.466878 1161 raft_consensus.cc:2241] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.467749 1161 raft_consensus.cc:2270] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.470674 1161 ts_tablet_manager.cc:1905] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.472236 1166 ts_tablet_manager.cc:1918] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.472590 1166 log.cc:1199] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3c8c317c10e14332a3b3919808e4a512
I20250625 07:52:19.473347 1166 ts_tablet_manager.cc:1939] T 3c8c317c10e14332a3b3919808e4a512 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:19.477263 1158 tablet_bootstrap.cc:654] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.482079 1542 tablet.cc:1722] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:19.490700 986 leader_election.cc:343] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9e77df5e7b3447fdbdb1157e74cf8b6d
I20250625 07:52:19.491523 986 leader_election.cc:304] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:19.493248 1485 raft_consensus.cc:2747] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:19.496412 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0657163b9a0d4f7f99f57f7fd1f32462 (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0657163b9a0d4f7f99f57f7fd1f32462 consensus state has no leader
I20250625 07:52:19.504122 1166 tablet_replica.cc:331] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:19.504874 1162 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.505643 1166 raft_consensus.cc:2241] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.506966 1162 ts_tablet_manager.cc:1397] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20250625 07:52:19.507745 1166 raft_consensus.cc:2270] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.508286 1536 raft_consensus.cc:491] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.508493 1536 raft_consensus.cc:513] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.509516 1536 leader_election.cc:290] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.510275 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.510860 851 leader_election.cc:343] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.510007 1545 tablet.cc:1722] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:19.513973 1166 ts_tablet_manager.cc:1905] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:19.365473 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 32a9f4cf6838411fab06fa6d35ae9492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a9f4cf6838411fab06fa6d35ae9492 already in progress: creating tablet
W20250625 07:52:19.515250 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.515304 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cbcfdbba5a84de1ac582e4687f347a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.515398 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feabe5a9dd014d698a57f45b19bad2cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.515221 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3221b04d095841d4a01f75c2a8004dce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.515331 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 149863us
W20250625 07:52:19.517131 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 871b603cb92249c6b0f16c825bf46232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871b603cb92249c6b0f16c825bf46232 already in progress: creating tablet
I20250625 07:52:19.517393 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a2517172003a44c492a3e3b9260bc0d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.517457 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ea62dfb8f20f4caeaa79ea1e85d99930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62dfb8f20f4caeaa79ea1e85d99930 already in progress: creating tablet
W20250625 07:52:19.518177 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 26d2bb8b0a8f45079083f2b42bc25011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26d2bb8b0a8f45079083f2b42bc25011 already in progress: creating tablet
I20250625 07:52:19.518339 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.518940 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6846d6c9101b41eba83b31905770dd57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.519366 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6846d6c9101b41eba83b31905770dd57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.520459 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3c8c317c10e14332a3b3919808e4a512 (table default.Table_ce2222183ca64261b57847f5be67d81a [id=152fbc75a19b4eacb4909704af909443]) successfully deleted
W20250625 07:52:19.521026 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a976166b59b44d52a920b760df58965c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.522645 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06acb0609d5e417886a27bc6693624ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.522845 885 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.523125 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:19.523309 850 leader_election.cc:343] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.523437 850 leader_election.cc:304] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.523677 1536 raft_consensus.cc:2747] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:19.523782 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 156524us
W20250625 07:52:19.524014 850 leader_election.cc:343] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.524152 850 leader_election.cc:304] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.524408 1536 raft_consensus.cc:2747] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.525105 1160 raft_consensus.cc:2241] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.525281 1160 raft_consensus.cc:2270] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.525838 1160 ts_tablet_manager.cc:1905] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.527447 1533 mvcc.cc:204] Tried to move back new op lower bound from 7171432198617309184 to 7171432195927900160. Current Snapshot: MvccSnapshot[applied={T|T < 7171432198617309184}]
W20250625 07:52:19.517961 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2517172003a44c492a3e3b9260bc0d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.514982 1162 raft_consensus.cc:357] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.533872 1162 raft_consensus.cc:383] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.533969 1162 raft_consensus.cc:738] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.539799 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_091524b607f045b985a03d229bc7c8e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.540333 713 leader_election.cc:343] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0452e73865964f709310fd22ac39a5f0
I20250625 07:52:19.540551 713 leader_election.cc:304] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.540650 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.541056 1506 raft_consensus.cc:2747] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:19.541352 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091524b607f045b985a03d229bc7c8e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.541507 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_abbbaaa7e3f6474bbb0003c420461b34" } new_table_name: "default.Table_913332b4f3224e12941ecc4c9d33f0d2" modify_external_catalogs: true
I20250625 07:52:19.545071 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_71387ac73a3149429c4f14fe3995a852" } new_table_name: "default.Table_69353bbfa60e4fc594bf8e56bd3f2b17" modify_external_catalogs: true
I20250625 07:52:19.549095 915 raft_consensus.cc:1273] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:19.550709 1543 consensus_queue.cc:1035] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:19.551448 1160 ts_tablet_manager.cc:1918] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.552171 1160 log.cc:1199] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3c0edbf6cbdb49a6b3d3fff7f347c8ce
I20250625 07:52:19.552592 1160 ts_tablet_manager.cc:1939] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.554188 1160 tablet_replica.cc:331] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:19.554544 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (table default.Table_ac06645d7b71464ab4e4e92402c14a13 [id=9511cff29b2441689b47351734929304]) successfully deleted
I20250625 07:52:19.554530 1160 raft_consensus.cc:2241] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.555063 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a6db9072062d4e5895af0868d99ac31b" } modify_external_catalogs: true
I20250625 07:52:19.555271 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a6db9072062d4e5895af0868d99ac31b" } modify_external_catalogs: true
I20250625 07:52:19.555222 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_78216911b4cc4ad2a1f0b223bf5edda8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.555552 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aacfa5f603c8417288db11b6ae09f880" } modify_external_catalogs: true
I20250625 07:52:19.555823 1160 raft_consensus.cc:2270] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.555872 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aacfa5f603c8417288db11b6ae09f880" } modify_external_catalogs: true
W20250625 07:52:19.555940 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e41e53efb6f342d2b1e4d642f7aaad10 consensus state has no leader
I20250625 07:52:19.556234 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7fc614d790ab4feaa0e3167c5de7cffd" } modify_external_catalogs: true
I20250625 07:52:19.556414 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7fc614d790ab4feaa0e3167c5de7cffd" } modify_external_catalogs: true
I20250625 07:52:19.558745 1542 tablet.cc:1717] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:19.559309 1162 consensus_queue.cc:260] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.556262 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_33aedac05c3649c0a21c0367c4ff59a3" } modify_external_catalogs: true
I20250625 07:52:19.565548 1543 consensus_queue.cc:1035] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:19.566386 1162 ts_tablet_manager.cc:1428] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20250625 07:52:19.566254 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_33aedac05c3649c0a21c0367c4ff59a3" } modify_external_catalogs: true
W20250625 07:52:19.570950 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65231ee1a59d49eb96ba352b74c3460c has no consensus state
I20250625 07:52:19.574654 1162 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:19.586433 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78216911b4cc4ad2a1f0b223bf5edda8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.565884 1536 raft_consensus.cc:491] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.588349 1536 raft_consensus.cc:513] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.588791 1166 ts_tablet_manager.cc:1918] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.588963 1166 log.cc:1199] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b769ac57346a427abd22214eebc0d324
I20250625 07:52:19.589195 1536 leader_election.cc:290] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.584096 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.590986 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:19.592588 850 leader_election.cc:343] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.592717 713 consensus_peers.cc:487] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:19.592806 850 leader_election.cc:304] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.593684 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 1eab049ad7fc4b5581e5c31a14491306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.593813 1162 tablet_bootstrap.cc:654] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.594838 1536 raft_consensus.cc:2747] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.594834 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7a2b94af59648239cfd1e5ce5e8b968" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.594137 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7a2b94af59648239cfd1e5ce5e8b968" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.596174 851 leader_election.cc:343] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.592914 1166 ts_tablet_manager.cc:1939] T b769ac57346a427abd22214eebc0d324 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:19.597101 850 leader_election.cc:343] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.597245 850 leader_election.cc:304] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.597720 1536 raft_consensus.cc:2747] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.600384 1166 tablet_replica.cc:331] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:19.600652 1166 raft_consensus.cc:2241] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.600919 1166 raft_consensus.cc:2270] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.601233 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_85d42dece3ab4be5a22696c28417f194" } new_table_name: "default.Table_8dc239a51cef42bd98cfd3a381d7b157" modify_external_catalogs: true
I20250625 07:52:19.601547 1166 ts_tablet_manager.cc:1905] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:19.608536 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2082e5e28c5c4d1fac7e40acf77f75e6 consensus state has no leader
I20250625 07:52:19.610790 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b769ac57346a427abd22214eebc0d324 (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:19.613129 1162 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.613340 1162 ts_tablet_manager.cc:1397] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.002s
I20250625 07:52:19.614290 1162 raft_consensus.cc:357] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.614557 1162 raft_consensus.cc:383] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.614694 1162 raft_consensus.cc:738] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.615797 1162 consensus_queue.cc:260] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.616366 1162 ts_tablet_manager.cc:1428] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:19.618505 1162 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.620894 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7bdec7af96c94b47a5b5c890625f02cc" } new_table_name: "default.Table_5a7fc7d8b43c4a7b805952d5fe0e01f7" modify_external_catalogs: true
I20250625 07:52:19.616544 1166 ts_tablet_manager.cc:1918] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.625392 1166 log.cc:1199] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0452e73865964f709310fd22ac39a5f0
I20250625 07:52:19.625731 1162 tablet_bootstrap.cc:654] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.625871 1166 ts_tablet_manager.cc:1939] T 0452e73865964f709310fd22ac39a5f0 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:19.626832 1166 tablet_replica.cc:331] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:19.627079 1166 raft_consensus.cc:2241] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.627272 1166 raft_consensus.cc:2270] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:19.628414 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4627f4da6b4542aff1e44157087b14 consensus state has no leader
I20250625 07:52:19.630858 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0452e73865964f709310fd22ac39a5f0 (table default.Table_c589a7d832eb4e7fb74f00c8fcbaa680 [id=7ce5c0adc972450bb89be40d18e4cc30]) successfully deleted
W20250625 07:52:19.631333 713 consensus_peers.cc:487] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:19.631563 1166 ts_tablet_manager.cc:1905] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.636554 1160 ts_tablet_manager.cc:1905] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.640969 1166 ts_tablet_manager.cc:1918] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.643031 1166 log.cc:1199] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/95a763abda074d358f176fd7e7be0e5c
I20250625 07:52:19.643477 1166 ts_tablet_manager.cc:1939] T 95a763abda074d358f176fd7e7be0e5c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:19.644481 1166 tablet_replica.cc:331] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:19.644728 1166 raft_consensus.cc:2241] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.644711 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 95a763abda074d358f176fd7e7be0e5c (table default.Table_1e843117005043efadef968ca11c482f [id=44ba0323288e45269c5dda7cd7f60f84]) successfully deleted
I20250625 07:52:19.644968 1166 raft_consensus.cc:2270] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.645841 1166 ts_tablet_manager.cc:1905] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.645939 1542 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:19.652714 1162 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.653026 1162 ts_tablet_manager.cc:1397] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20250625 07:52:19.653920 1162 raft_consensus.cc:357] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.654174 1162 raft_consensus.cc:383] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.654301 1162 raft_consensus.cc:738] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.654578 1162 consensus_queue.cc:260] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.655169 1162 ts_tablet_manager.cc:1428] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:19.655529 1162 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.657871 1162 tablet_bootstrap.cc:654] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.660069 1166 ts_tablet_manager.cc:1918] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.660224 1166 log.cc:1199] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7a53d8f50224460b932f4f44659336da
I20250625 07:52:19.660661 1166 ts_tablet_manager.cc:1939] T 7a53d8f50224460b932f4f44659336da P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:19.663269 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7a53d8f50224460b932f4f44659336da (table default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]) successfully deleted
I20250625 07:52:19.669193 1158 tablet_bootstrap.cc:492] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:19.675872 1158 ts_tablet_manager.cc:1397] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.283s user 0.003s sys 0.006s
I20250625 07:52:19.674794 1556 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:19.681964 1163 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.682267 1163 ts_tablet_manager.cc:1397] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.399s user 0.033s sys 0.075s
I20250625 07:52:19.685447 1163 raft_consensus.cc:357] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.686370 1163 raft_consensus.cc:383] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.686754 1163 raft_consensus.cc:738] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.687793 1163 consensus_queue.cc:260] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.689641 1163 ts_tablet_manager.cc:1428] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250625 07:52:19.690898 1163 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.692960 1162 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.693161 1162 ts_tablet_manager.cc:1397] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.012s
I20250625 07:52:19.696411 1162 raft_consensus.cc:357] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.699505 1162 raft_consensus.cc:383] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.699429 1163 tablet_bootstrap.cc:654] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.699682 1162 raft_consensus.cc:738] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.700069 1158 raft_consensus.cc:357] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.700362 1158 raft_consensus.cc:383] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.701043 1162 consensus_queue.cc:260] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.704044 1162 ts_tablet_manager.cc:1428] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:19.705502 1162 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:19.706800 1543 raft_consensus.cc:491] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.707142 1543 raft_consensus.cc:513] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.708021 1543 leader_election.cc:290] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.709340 1162 tablet_bootstrap.cc:654] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.713905 1163 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:19.714107 713 leader_election.cc:343] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e76df9405d91488dad0cfa7f8755bb1c
I20250625 07:52:19.714542 1163 ts_tablet_manager.cc:1397] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.006s
I20250625 07:52:19.718463 1158 raft_consensus.cc:738] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.718694 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6885e8b33124109bb5ab1f1128522de" } new_table_name: "default.Table_4bfac23414d34778bd98d79dcf630b42" modify_external_catalogs: true
I20250625 07:52:19.719028 1161 ts_tablet_manager.cc:1918] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.719074 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e76df9405d91488dad0cfa7f8755bb1c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.719187 1161 log.cc:1199] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/365ef91e7bac43e283602d3d036853ff
I20250625 07:52:19.719631 1161 ts_tablet_manager.cc:1939] T 365ef91e7bac43e283602d3d036853ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.719604 1010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47290 (request call id 82) took 1161 ms. Trace:
I20250625 07:52:19.719738 1010 rpcz_store.cc:276] 0625 07:52:18.557119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.557180 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:19.718382 (+1161202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 07:52:19.720029 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 391) took 1890 ms. Trace:
I20250625 07:52:19.720144 1161 rpcz_store.cc:276] 0625 07:52:17.829984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.830277 (+ 293us) service_pool.cc:225] Handling call
0625 07:52:19.465387 (+1635110us) ts_tablet_manager.cc:1145] Deleting tablet 365ef91e7bac43e283602d3d036853ff
0625 07:52:19.465410 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.687637 (+222227us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.703349 (+ 15712us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.719987 (+ 16638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634179}
I20250625 07:52:19.720479 1161 tablet_replica.cc:331] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.720676 1161 raft_consensus.cc:2241] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.720885 1161 raft_consensus.cc:2270] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.721484 1161 ts_tablet_manager.cc:1905] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.717639 1163 raft_consensus.cc:357] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.722220 1163 raft_consensus.cc:383] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.722616 1163 raft_consensus.cc:738] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.723474 1163 consensus_queue.cc:260] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.725452 1163 ts_tablet_manager.cc:1428] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250625 07:52:19.726035 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 365ef91e7bac43e283602d3d036853ff (table default.Table_339dbe043bad47148d466ca915d84d33 [id=e67e3e600b7c42178b012fdec003c3b3]) successfully deleted
I20250625 07:52:19.726603 1163 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:19.729377 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61e076870efc4744b1224ddcb0eef3fa consensus state has no leader
W20250625 07:52:19.727572 713 leader_election.cc:343] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.729830 713 leader_election.cc:304] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.734357 1163 tablet_bootstrap.cc:654] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.741000 1536 raft_consensus.cc:491] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.741233 1536 raft_consensus.cc:513] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.742116 1536 leader_election.cc:290] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.743271 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7100ef72ff14c24be92c21c6eb8a9de" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.743952 851 leader_election.cc:343] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.744738 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7100ef72ff14c24be92c21c6eb8a9de" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:19.744962 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2a5dd2a80df94b488a9c2ed687499a4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.745342 850 leader_election.cc:343] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.745468 850 leader_election.cc:304] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:19.745582 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a5dd2a80df94b488a9c2ed687499a4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.745886 1536 raft_consensus.cc:2747] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.746887 1543 raft_consensus.cc:2747] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.747254 1158 consensus_queue.cc:260] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.747680 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_404e6f3b9acf49a18305dbe0b2cac534"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.748199 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_404e6f3b9acf49a18305dbe0b2cac534 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.749469 1158 ts_tablet_manager.cc:1428] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.072s user 0.001s sys 0.001s
I20250625 07:52:19.750679 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f8c817fc80f64e76a3078837e90e97ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.751194 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c817fc80f64e76a3078837e90e97ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.753489 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 1898ff5b7626478bb997f7a433477498 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.755968 1161 ts_tablet_manager.cc:1918] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.763224 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b36659e6e06d41298f3a614061f97b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.764901 1161 log.cc:1199] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/86ea9df068a94d9187f2f9620e371a39
I20250625 07:52:19.765038 1158 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:19.764683 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b36659e6e06d41298f3a614061f97b73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.766409 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1860ecd4dd0340c39868730e3b641d4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.769001 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1860ecd4dd0340c39868730e3b641d4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.780488 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_322b342d0f384574a0239ed582647def"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.782668 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_322b342d0f384574a0239ed582647def in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.784744 1161 ts_tablet_manager.cc:1939] T 86ea9df068a94d9187f2f9620e371a39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.786542 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 392) took 1946 ms. Trace:
I20250625 07:52:19.786677 1161 rpcz_store.cc:276] 0625 07:52:17.839186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.839240 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:19.720405 (+1881165us) ts_tablet_manager.cc:1145] Deleting tablet 86ea9df068a94d9187f2f9620e371a39
0625 07:52:19.720414 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.724813 (+ 4399us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.755840 (+ 31027us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.785372 (+ 29532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880948}
I20250625 07:52:19.787335 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 86ea9df068a94d9187f2f9620e371a39 (table default.Table_17d5751097fa40feaf90d0fb3c97e1b5 [id=5281aca609cf421990f96de3424b4442]) successfully deleted
I20250625 07:52:19.781757 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dfa7845282e043bd8a70386031f91ff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.788970 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfa7845282e043bd8a70386031f91ff7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.790866 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b1eaf0266c04873b185c1c0c2da201c with delete_type TABLET_DATA_DELETED (replaced by tablet 445bfed532004108a12c8fd78370c1cb) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:19.793345 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 393) took 1920 ms. Trace:
I20250625 07:52:19.793548 1161 rpcz_store.cc:276] 0625 07:52:17.867552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.867611 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:19.787123 (+1919512us) ts_tablet_manager.cc:1145] Deleting tablet 23af43b9ceeb4f2b812ee251ea8545ce
0625 07:52:19.787134 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.788100 (+ 966us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919298}
W20250625 07:52:19.794449 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af43b9ceeb4f2b812ee251ea8545ce already in progress: creating tablet
I20250625 07:52:19.796978 1158 tablet_bootstrap.cc:654] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.799700 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3de45b2a2d3049d788bc30869bd75d2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.801064 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3de45b2a2d3049d788bc30869bd75d2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.805049 1545 tablet.cc:1717] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:19.813361 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 395) took 1937 ms. Trace:
I20250625 07:52:19.813504 1161 rpcz_store.cc:276] 0625 07:52:17.875468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.875526 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:19.794071 (+1918545us) ts_tablet_manager.cc:1145] Deleting tablet 2bcfe7c3c54741fbb0aca69650fc8eee
0625 07:52:19.811090 (+ 17019us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.813035 (+ 1945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1918298}
W20250625 07:52:19.816025 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bcfe7c3c54741fbb0aca69650fc8eee already in progress: creating tablet
I20250625 07:52:19.820487 1163 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:19.820886 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 394) took 1945 ms. Trace:
I20250625 07:52:19.820997 1161 rpcz_store.cc:276] 0625 07:52:17.875313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.875525 (+ 212us) service_pool.cc:225] Handling call
0625 07:52:19.816349 (+1940824us) ts_tablet_manager.cc:1145] Deleting tablet 29eaa8d93e5b4d1d8ed147b60c7da859
0625 07:52:19.820550 (+ 4201us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.820676 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940444}
W20250625 07:52:19.821645 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29eaa8d93e5b4d1d8ed147b60c7da859 already in progress: creating tablet
I20250625 07:52:19.822000 1163 ts_tablet_manager.cc:1397] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.007s
I20250625 07:52:19.822636 1536 raft_consensus.cc:491] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.822801 1536 raft_consensus.cc:513] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.823037 1163 raft_consensus.cc:357] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.823405 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 396) took 1937 ms. Trace:
I20250625 07:52:19.823513 1161 rpcz_store.cc:276] 0625 07:52:17.883762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.884018 (+ 256us) service_pool.cc:225] Handling call
0625 07:52:19.821258 (+1937240us) ts_tablet_manager.cc:1145] Deleting tablet 23af43b9ceeb4f2b812ee251ea8545ce
0625 07:52:19.821267 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.821349 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937028}
I20250625 07:52:19.823904 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 398) took 1937 ms. Trace:
I20250625 07:52:19.824054 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6606a9559e6b47d28669083af5aee5bd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:19.824513 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 23af43b9ceeb4f2b812ee251ea8545ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af43b9ceeb4f2b812ee251ea8545ce already in progress: creating tablet
I20250625 07:52:19.824005 1161 rpcz_store.cc:276] 0625 07:52:17.886643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.886698 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:19.823732 (+1937034us) ts_tablet_manager.cc:1145] Deleting tablet 2bcfe7c3c54741fbb0aca69650fc8eee
0625 07:52:19.823742 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.823843 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936866}
I20250625 07:52:19.825127 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 397) took 1940 ms. Trace:
I20250625 07:52:19.825227 1161 rpcz_store.cc:276] 0625 07:52:17.884880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.887931 (+ 3051us) service_pool.cc:225] Handling call
0625 07:52:19.824965 (+1937034us) ts_tablet_manager.cc:1145] Deleting tablet 716e0c4dab584788bbb7631b4ca934bf
0625 07:52:19.824974 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.825072 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1936831}
I20250625 07:52:19.825361 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6606a9559e6b47d28669083af5aee5bd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:19.825556 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 400) took 1912 ms. Trace:
I20250625 07:52:19.825641 1161 rpcz_store.cc:276] 0625 07:52:17.913474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.913540 (+ 66us) service_pool.cc:225] Handling call
0625 07:52:19.825421 (+1911881us) ts_tablet_manager.cc:1145] Deleting tablet 9418b81f010d4f369d3409e9364bc5fa
0625 07:52:19.825429 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.825495 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911571}
I20250625 07:52:19.825945 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 401) took 1912 ms. Trace:
W20250625 07:52:19.826382 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2bcfe7c3c54741fbb0aca69650fc8eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bcfe7c3c54741fbb0aca69650fc8eee already in progress: creating tablet
W20250625 07:52:19.826723 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9418b81f010d4f369d3409e9364bc5fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9418b81f010d4f369d3409e9364bc5fa already in progress: creating tablet
I20250625 07:52:19.827095 1161 rpcz_store.cc:276] 0625 07:52:17.913694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:17.913767 (+ 73us) service_pool.cc:225] Handling call
0625 07:52:19.825817 (+1912050us) ts_tablet_manager.cc:1145] Deleting tablet b2bb060cdd9a48688967989d7491cfc1
0625 07:52:19.825824 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.825891 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911829}
W20250625 07:52:19.827495 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b2bb060cdd9a48688967989d7491cfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2bb060cdd9a48688967989d7491cfc1 already in progress: creating tablet
I20250625 07:52:19.823331 1163 raft_consensus.cc:383] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.828027 1163 raft_consensus.cc:738] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.828207 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 409) took 1728 ms. Trace:
I20250625 07:52:19.828325 1161 rpcz_store.cc:276] 0625 07:52:18.099612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.099830 (+ 218us) service_pool.cc:225] Handling call
0625 07:52:19.827741 (+1727911us) ts_tablet_manager.cc:1145] Deleting tablet b5f917cfd2384a72920847f8b89358b9
0625 07:52:19.827750 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.827885 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726660}
W20250625 07:52:19.828886 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b5f917cfd2384a72920847f8b89358b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5f917cfd2384a72920847f8b89358b9 already in progress: creating tablet
I20250625 07:52:19.830350 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 410) took 1730 ms. Trace:
I20250625 07:52:19.830472 1161 rpcz_store.cc:276] 0625 07:52:18.099764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.099940 (+ 176us) service_pool.cc:225] Handling call
0625 07:52:19.829158 (+1729218us) ts_tablet_manager.cc:1145] Deleting tablet 9cc1842f257d409ba0a56873f582f023
0625 07:52:19.829166 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.829964 (+ 798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727966}
W20250625 07:52:19.830869 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9cc1842f257d409ba0a56873f582f023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cc1842f257d409ba0a56873f582f023 already in progress: creating tablet
I20250625 07:52:19.823556 1536 leader_election.cc:290] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:19.824504 850 leader_election.cc:343] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:19.831207 851 leader_election.cc:343] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.830179 1163 consensus_queue.cc:260] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.831485 851 leader_election.cc:304] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.831717 1163 ts_tablet_manager.cc:1428] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 07:52:19.832067 1163 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:19.832139 1308 raft_consensus.cc:2747] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.833907 1163 tablet_bootstrap.cc:654] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.837209 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 412) took 1691 ms. Trace:
I20250625 07:52:19.837096 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c43bae11f964428a83cb53dc9dee5bdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.837337 1161 rpcz_store.cc:276] 0625 07:52:18.139664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.139719 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:19.831144 (+1691425us) ts_tablet_manager.cc:1145] Deleting tablet a33b7cab19244ec08d3821b6cc4c072d
0625 07:52:19.831155 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.831232 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691131}
I20250625 07:52:19.837832 1161 tablet_replica.cc:331] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.838029 1161 raft_consensus.cc:2241] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:19.838042 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a33b7cab19244ec08d3821b6cc4c072d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a33b7cab19244ec08d3821b6cc4c072d already in progress: creating tablet
I20250625 07:52:19.838279 1161 raft_consensus.cc:2270] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.838215 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8dc239a51cef42bd98cfd3a381d7b157" } new_table_name: "default.Table_0a167e3ae60a42e7a47dca60a52078f8" modify_external_catalogs: true
W20250625 07:52:19.838747 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c43bae11f964428a83cb53dc9dee5bdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.838830 1161 ts_tablet_manager.cc:1905] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.847028 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]). Expected version 2 got 0
I20250625 07:52:19.848047 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a] alter complete (version 2)
I20250625 07:52:19.848909 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_78216911b4cc4ad2a1f0b223bf5edda8" } new_table_name: "default.Table_d50dc89e611a4cd490d628b4a824a34f" modify_external_catalogs: true
I20250625 07:52:19.852520 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0319c343efd4139bb3d83e99569aa6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:19.854657 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 (DEFAULT_TABLE table=default.Table_29b24dec640147299371e6748d35cc5b [id=6041ac7669374b268f40cadc19984959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.854957 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b5bc347c278c44c69444caf9ded72e1a (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.855166 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9578d77db00242d48f4e8f6f8263ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.855362 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bc347c278c44c69444caf9ded72e1a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:19.856024 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0319c343efd4139bb3d83e99569aa6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.856274 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.857054 754 tablet_service.cc:1468] Processing CreateTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 (DEFAULT_TABLE table=default.Table_29b24dec640147299371e6748d35cc5b [id=6041ac7669374b268f40cadc19984959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.857431 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9578d77db00242d48f4e8f6f8263ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.857877 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 506794253c84486faa926de885c48c58 (DEFAULT_TABLE table=default.Table_77bd933ec7f6471886c2b3454654fe9c [id=a3fa61e4bf994ec2b82fa32ede09fa0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.858263 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506794253c84486faa926de885c48c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.859898 1012 tablet_service.cc:1468] Processing CreateTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.860266 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06ecad6c7464ffe8ec6f632634307c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.864363 752 tablet_service.cc:1468] Processing CreateTablet for tablet 506794253c84486faa926de885c48c58 (DEFAULT_TABLE table=default.Table_77bd933ec7f6471886c2b3454654fe9c [id=a3fa61e4bf994ec2b82fa32ede09fa0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.864863 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506794253c84486faa926de885c48c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.864913 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 35a93e9198d1417bb1b8e00560921483 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.854657 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 (DEFAULT_TABLE table=default.Table_f3cd40e7670e492bba639148deda5ffd [id=493afd31fad94b3586421e8f7fbde02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.865883 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c584e6bffb4f0a8020dadf78d718a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.868934 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a93e9198d1417bb1b8e00560921483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.869362 757 tablet_service.cc:1468] Processing CreateTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 (DEFAULT_TABLE table=default.Table_f3cd40e7670e492bba639148deda5ffd [id=493afd31fad94b3586421e8f7fbde02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.869745 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c584e6bffb4f0a8020dadf78d718a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.873670 746 tablet_service.cc:1468] Processing CreateTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.874097 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06ecad6c7464ffe8ec6f632634307c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.874166 1161 ts_tablet_manager.cc:1918] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.874310 1161 log.cc:1199] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/23755f70822740e08ca6a5e5b5571458
I20250625 07:52:19.874763 1161 ts_tablet_manager.cc:1939] T 23755f70822740e08ca6a5e5b5571458 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.877789 751 tablet_service.cc:1468] Processing CreateTablet for tablet b5bc347c278c44c69444caf9ded72e1a (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.878163 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bc347c278c44c69444caf9ded72e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.881476 889 tablet_service.cc:1468] Processing CreateTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 (DEFAULT_TABLE table=default.Table_29b24dec640147299371e6748d35cc5b [id=6041ac7669374b268f40cadc19984959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.881908 885 tablet_service.cc:1468] Processing CreateTablet for tablet 506794253c84486faa926de885c48c58 (DEFAULT_TABLE table=default.Table_77bd933ec7f6471886c2b3454654fe9c [id=a3fa61e4bf994ec2b82fa32ede09fa0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.882267 892 tablet_service.cc:1468] Processing CreateTablet for tablet 35a93e9198d1417bb1b8e00560921483 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.882524 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.882727 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.883188 886 tablet_service.cc:1468] Processing CreateTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 (DEFAULT_TABLE table=default.Table_f3cd40e7670e492bba639148deda5ffd [id=493afd31fad94b3586421e8f7fbde02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.883430 1160 ts_tablet_manager.cc:1918] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.883631 1160 log.cc:1199] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/59df0f66116e4dfd9386d107cab0aaf6
I20250625 07:52:19.883693 883 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.884366 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9578d77db00242d48f4e8f6f8263ef09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.888243 1160 ts_tablet_manager.cc:1939] T 59df0f66116e4dfd9386d107cab0aaf6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:19.888918 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:19.889348 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 414) took 1705 ms. Trace:
I20250625 07:52:19.889338 894 tablet_service.cc:1468] Processing CreateTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.890184 1158 tablet_bootstrap.cc:492] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:19.890355 1158 ts_tablet_manager.cc:1397] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.017s
I20250625 07:52:19.891155 1158 raft_consensus.cc:357] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.891386 1158 raft_consensus.cc:383] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.891466 1158 raft_consensus.cc:738] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.891708 1158 consensus_queue.cc:260] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.892204 1158 ts_tablet_manager.cc:1428] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:19.889467 1161 rpcz_store.cc:276] 0625 07:52:18.183859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.184378 (+ 519us) service_pool.cc:225] Handling call
0625 07:52:19.837638 (+1653260us) ts_tablet_manager.cc:1145] Deleting tablet 23755f70822740e08ca6a5e5b5571458
0625 07:52:19.837651 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.843124 (+ 5473us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.874008 (+ 30884us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.889309 (+ 15301us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13698,"tablet-delete.queue_time_us":1615653}
I20250625 07:52:19.894891 1158 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:19.885273 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7a53d8f50224460b932f4f44659336da because the tablet was not found. No further retry: Not found: Tablet not found: 7a53d8f50224460b932f4f44659336da
I20250625 07:52:19.900424 1158 tablet_bootstrap.cc:654] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.898682 1161 tablet_replica.cc:331] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.901377 1161 raft_consensus.cc:2241] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.901572 1161 raft_consensus.cc:2270] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.894277 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 23755f70822740e08ca6a5e5b5571458 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
I20250625 07:52:19.902153 1161 ts_tablet_manager.cc:1905] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.902670 883 tablet_service.cc:1468] Processing CreateTablet for tablet b5bc347c278c44c69444caf9ded72e1a (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:19.897042 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2082e5e28c5c4d1fac7e40acf77f75e6 consensus state has no leader
I20250625 07:52:19.904163 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 0102a3b176e146a8b19abe22d3aba656 (DEFAULT_TABLE table=default.Table_f4184bdee984426ea3612ca34ca1bf92 [id=2a743c4696284539941ed3c1309a1660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.905699 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:19.910080 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0102a3b176e146a8b19abe22d3aba656 (DEFAULT_TABLE table=default.Table_f4184bdee984426ea3612ca34ca1bf92 [id=2a743c4696284539941ed3c1309a1660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.910398 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:19.910534 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0102a3b176e146a8b19abe22d3aba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.915426 1161 ts_tablet_manager.cc:1918] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.915586 1161 log.cc:1199] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9e77df5e7b3447fdbdb1157e74cf8b6d
I20250625 07:52:19.915988 1161 ts_tablet_manager.cc:1939] T 9e77df5e7b3447fdbdb1157e74cf8b6d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.916210 745 tablet_service.cc:1468] Processing CreateTablet for tablet e35d77471f7d4866a8c30324b5da0170 (DEFAULT_TABLE table=default.Table_c65ada52ed03480196c90fc1cd9f3bbb [id=53084dbbc7114c668a9fc9bbe33780a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.916617 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35d77471f7d4866a8c30324b5da0170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.916648 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06ecad6c7464ffe8ec6f632634307c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.920984 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506794253c84486faa926de885c48c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.924944 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a93e9198d1417bb1b8e00560921483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.925289 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 47c9aca996884ffeab88381dd3cb189b (DEFAULT_TABLE table=default.Table_7e8eb7e9baf5497ca408e7592b57e4dc [id=53a8409531d94cc7aa5c8ecb4f8e1894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.925707 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9aca996884ffeab88381dd3cb189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.925649 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0102a3b176e146a8b19abe22d3aba656 (DEFAULT_TABLE table=default.Table_f4184bdee984426ea3612ca34ca1bf92 [id=2a743c4696284539941ed3c1309a1660]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.926015 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0102a3b176e146a8b19abe22d3aba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.929474 888 tablet_service.cc:1468] Processing CreateTablet for tablet 47c9aca996884ffeab88381dd3cb189b (DEFAULT_TABLE table=default.Table_7e8eb7e9baf5497ca408e7592b57e4dc [id=53a8409531d94cc7aa5c8ecb4f8e1894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.929874 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9aca996884ffeab88381dd3cb189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.930872 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c584e6bffb4f0a8020dadf78d718a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.931274 1517 catalog_manager.cc:1261] Loaded cluster ID: f1c1a60bf7df437d8f4c2e6d554d8ac6
I20250625 07:52:19.931377 1517 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: loading cluster ID for follower catalog manager: success
I20250625 07:52:19.933285 893 tablet_service.cc:1468] Processing CreateTablet for tablet 65231ee1a59d49eb96ba352b74c3460c (DEFAULT_TABLE table=default.Table_69353bbfa60e4fc594bf8e56bd3f2b17 [id=dae978f4a15b4e4ba338dd62912614b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.933717 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65231ee1a59d49eb96ba352b74c3460c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.945896 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 65231ee1a59d49eb96ba352b74c3460c (DEFAULT_TABLE table=default.Table_69353bbfa60e4fc594bf8e56bd3f2b17 [id=dae978f4a15b4e4ba338dd62912614b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.946406 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65231ee1a59d49eb96ba352b74c3460c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.950218 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bc347c278c44c69444caf9ded72e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.951865 757 tablet_service.cc:1468] Processing CreateTablet for tablet 65231ee1a59d49eb96ba352b74c3460c (DEFAULT_TABLE table=default.Table_69353bbfa60e4fc594bf8e56bd3f2b17 [id=dae978f4a15b4e4ba338dd62912614b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.952297 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65231ee1a59d49eb96ba352b74c3460c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.957015 1536 raft_consensus.cc:491] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:19.957209 1536 raft_consensus.cc:513] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:19.957908 1536 leader_election.cc:290] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:19.958571 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a9945f38ef148caac59acaadcb2ef4c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:19.959168 851 leader_election.cc:343] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.961071 746 tablet_service.cc:1468] Processing CreateTablet for tablet 47c9aca996884ffeab88381dd3cb189b (DEFAULT_TABLE table=default.Table_7e8eb7e9baf5497ca408e7592b57e4dc [id=53a8409531d94cc7aa5c8ecb4f8e1894]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.961467 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47c9aca996884ffeab88381dd3cb189b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.962164 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a9945f38ef148caac59acaadcb2ef4c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:19.962841 850 leader_election.cc:343] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:19.963001 850 leader_election.cc:304] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:19.963308 1536 raft_consensus.cc:2747] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:19.966009 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0102a3b176e146a8b19abe22d3aba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.966557 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 416) took 1707 ms. Trace:
I20250625 07:52:19.966683 1161 rpcz_store.cc:276] 0625 07:52:18.259058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.259193 (+ 135us) service_pool.cc:225] Handling call
0625 07:52:19.892546 (+1633353us) ts_tablet_manager.cc:1145] Deleting tablet 9e77df5e7b3447fdbdb1157e74cf8b6d
0625 07:52:19.898595 (+ 6049us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.905430 (+ 6835us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.915294 (+ 9864us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.966341 (+ 51047us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6016,"tablet-delete.queue_time_us":1629931}
I20250625 07:52:19.967075 1161 tablet_replica.cc:331] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:19.967146 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9e77df5e7b3447fdbdb1157e74cf8b6d (table default.Table_0d58b443a7b2466488c13f5ec50e1a0f [id=2ea294f53b78433f969900426f24ab54]) successfully deleted
I20250625 07:52:19.967271 1161 raft_consensus.cc:2241] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:19.967465 1161 raft_consensus.cc:2270] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:19.968039 1161 ts_tablet_manager.cc:1905] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:19.968065 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 59df0f66116e4dfd9386d107cab0aaf6 (table default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]) successfully deleted
I20250625 07:52:19.970293 754 tablet_service.cc:1468] Processing CreateTablet for tablet 35a93e9198d1417bb1b8e00560921483 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.970321 1011 tablet_service.cc:1468] Processing CreateTablet for tablet e35d77471f7d4866a8c30324b5da0170 (DEFAULT_TABLE table=default.Table_c65ada52ed03480196c90fc1cd9f3bbb [id=53084dbbc7114c668a9fc9bbe33780a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.970695 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a93e9198d1417bb1b8e00560921483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.970721 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35d77471f7d4866a8c30324b5da0170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.973278 1162 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:19.973438 1162 ts_tablet_manager.cc:1397] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.268s user 0.014s sys 0.037s
I20250625 07:52:19.974295 1162 raft_consensus.cc:357] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.974509 1162 raft_consensus.cc:383] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:19.974593 1162 raft_consensus.cc:738] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:19.974701 1161 ts_tablet_manager.cc:1918] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:19.974818 1161 log.cc:1199] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4fdb6a689ce6440ab7037484e2b9cdc5
I20250625 07:52:19.974869 1162 consensus_queue.cc:260] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:19.975358 1162 ts_tablet_manager.cc:1428] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:19.975728 1162 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:19.977573 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 32a9f4cf6838411fab06fa6d35ae9492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a9f4cf6838411fab06fa6d35ae9492 already in progress: creating tablet
W20250625 07:52:19.977830 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 871b603cb92249c6b0f16c825bf46232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871b603cb92249c6b0f16c825bf46232 already in progress: creating tablet
W20250625 07:52:19.978055 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ea62dfb8f20f4caeaa79ea1e85d99930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62dfb8f20f4caeaa79ea1e85d99930 already in progress: creating tablet
I20250625 07:52:19.978104 1162 tablet_bootstrap.cc:654] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:19.979542 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3bed3f04c60249aaa389c26788974f58
I20250625 07:52:19.979928 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:19.980527 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_77bd933ec7f6471886c2b3454654fe9c" } new_table_name: "default.Table_9caecbdf269849638e585aac6422de32" modify_external_catalogs: true
I20250625 07:52:19.981611 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f16f5e9c3f6d4110baa383f8551d2f91" } new_table_name: "default.Table_3910e10f6f4b41eeabfd11ac6c4343c7" modify_external_catalogs: true
I20250625 07:52:19.982787 1161 ts_tablet_manager.cc:1939] T 4fdb6a689ce6440ab7037484e2b9cdc5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:19.982957 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:19.983464 590 master_service.cc:948] ReplaceTablet: received request to replace tablet c263b671be384cd097041c60306034de from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.983989 590 master_service.cc:948] ReplaceTablet: received request to replace tablet de9ccb0a2d3846228acbc2769786a059 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.984347 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.984381 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 651c74bf9a9b47d6ab5a875a5c304783 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.984891 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3221b04d095841d4a01f75c2a8004dce" } new_table_name: "default.Table_e6092e518ceb4af7a5d84435008334ba" modify_external_catalogs: true
I20250625 07:52:19.985029 744 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:19.985618 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5179a144a9a0445f933ca02e79bbc122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5179a144a9a0445f933ca02e79bbc122 already in progress: creating tablet
I20250625 07:52:19.985908 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 96a1f823fd3b47c9b6d1cf5647f9a68a
I20250625 07:52:19.986599 594 master_service.cc:948] ReplaceTablet: received request to replace tablet f191436f3b9244eaa12d211b6f125175 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:19.986778 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:19.987527 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet faf049714a7e4d7eb947c29f272bbb7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faf049714a7e4d7eb947c29f272bbb7c already in progress: creating tablet
I20250625 07:52:19.988090 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 76fe692067954856908f936ced332c18
I20250625 07:52:19.988140 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.988842 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:19.988937 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d3a5e89e18914595b2f605c893c2d050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.989681 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3a5e89e18914595b2f605c893c2d050 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.990113 1517 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: acquiring CA information for follower catalog manager: success
I20250625 07:52:19.990041 895 tablet_service.cc:1468] Processing CreateTablet for tablet e35d77471f7d4866a8c30324b5da0170 (DEFAULT_TABLE table=default.Table_c65ada52ed03480196c90fc1cd9f3bbb [id=53084dbbc7114c668a9fc9bbe33780a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:19.990671 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e35d77471f7d4866a8c30324b5da0170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.991276 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1898ff5b7626478bb997f7a433477498
I20250625 07:52:19.991819 589 catalog_manager.cc:6652] ReplaceTablet: tablet 1898ff5b7626478bb997f7a433477498 deleted and replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75
I20250625 07:52:19.992237 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_85849694e69e4737a9793e7c46e40a87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.992920 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85849694e69e4737a9793e7c46e40a87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:19.994630 751 tablet_service.cc:1468] Processing CreateTablet for tablet 813f9f9fccc34685b73847c9d590ceb6 (DEFAULT_TABLE table=default.Table_adb5d0acb48c45b781c468e36e59b4c0 [id=3f984673a2964bf19fdbb95a9da4e109]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:19.994851 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 670f3b7f0a904656af418d01ed6f58b4 has no consensus state
I20250625 07:52:19.995244 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f9f9fccc34685b73847c9d590ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:19.995318 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f9279117557549b3826d8056f0d22581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:19.996001 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9279117557549b3826d8056f0d22581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:19.999142 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a1f823fd3b47c9b6d1cf5647f9a68a already in progress: creating tablet
W20250625 07:52:19.999619 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3bed3f04c60249aaa389c26788974f58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed3f04c60249aaa389c26788974f58 already in progress: creating tablet
I20250625 07:52:20.000348 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.000465 1162 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.000612 1162 ts_tablet_manager.cc:1397] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
W20250625 07:52:20.001226 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1898ff5b7626478bb997f7a433477498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1898ff5b7626478bb997f7a433477498 already in progress: creating tablet
I20250625 07:52:20.001555 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 76fe692067954856908f936ced332c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.001652 1162 raft_consensus.cc:357] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.001889 1162 raft_consensus.cc:383] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.001845 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea01ec621c614388990c5fddddbd17a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.001987 1162 raft_consensus.cc:738] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.002373 1162 consensus_queue.cc:260] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
W20250625 07:52:20.002547 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea01ec621c614388990c5fddddbd17a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.004874 885 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.005163 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:20.008924 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 76fe692067954856908f936ced332c18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76fe692067954856908f936ced332c18 already in progress: creating tablet
I20250625 07:52:20.010696 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_0ef24d5b4b124360b654d6903b994151 [id=23322aa9321140c39272b5b2a102fb02] alter complete (version 2)
I20250625 07:52:20.012890 1543 raft_consensus.cc:491] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.013080 1543 raft_consensus.cc:513] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.013695 1543 leader_election.cc:290] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.016650 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.016970 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 76fe692067954856908f936ced332c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.017805 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.018047 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.018771 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.018966 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.020627 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.020920 1166 tablet_replica.cc:331] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.021122 1166 raft_consensus.cc:2241] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.021307 1166 raft_consensus.cc:2270] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.021644 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7e46f27344b848318dcd1e782819b75e (DEFAULT_TABLE table=default.Table_e35c55dc42af4bb28966deeb87d520c8 [id=4fff90a74052472bb7bb8778a9786b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.022068 1166 ts_tablet_manager.cc:1905] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.022188 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e46f27344b848318dcd1e782819b75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.022805 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.025045 889 tablet_service.cc:1468] Processing CreateTablet for tablet 813f9f9fccc34685b73847c9d590ceb6 (DEFAULT_TABLE table=default.Table_adb5d0acb48c45b781c468e36e59b4c0 [id=3f984673a2964bf19fdbb95a9da4e109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.025570 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f9f9fccc34685b73847c9d590ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.027065 1162 ts_tablet_manager.cc:1428] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250625 07:52:20.028298 1162 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.028595 1166 ts_tablet_manager.cc:1918] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.028789 1166 log.cc:1199] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4c3c180785b64f7d85863c14cfd09ccc
I20250625 07:52:20.029191 1166 ts_tablet_manager.cc:1939] T 4c3c180785b64f7d85863c14cfd09ccc P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.032773 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.033277 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.033689 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.033898 1160 tablet_replica.cc:331] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:20.034224 1160 raft_consensus.cc:2241] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.034211 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7e46f27344b848318dcd1e782819b75e (DEFAULT_TABLE table=default.Table_e35c55dc42af4bb28966deeb87d520c8 [id=4fff90a74052472bb7bb8778a9786b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.034489 1160 raft_consensus.cc:2270] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.034596 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e46f27344b848318dcd1e782819b75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.035228 1160 ts_tablet_manager.cc:1905] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.041507 1166 tablet_replica.cc:331] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.041800 1166 raft_consensus.cc:2241] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.042037 1166 raft_consensus.cc:2270] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.042992 1166 ts_tablet_manager.cc:1905] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.045433 1162 tablet_bootstrap.cc:654] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.046985 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4c3c180785b64f7d85863c14cfd09ccc (table default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]) successfully deleted
I20250625 07:52:20.047482 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1f771628379842989b9b2099de65e2a0 (DEFAULT_TABLE table=default.Table_aab39ebeac1742cda147feb57cf092d7 [id=63a78517d1cb4d6790a9422cb7b822aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.047910 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f771628379842989b9b2099de65e2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.049504 1163 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.049736 1163 ts_tablet_manager.cc:1397] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.218s user 0.003s sys 0.001s
I20250625 07:52:20.050674 1163 raft_consensus.cc:357] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.050946 1163 raft_consensus.cc:383] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.051072 1163 raft_consensus.cc:738] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.051370 1163 consensus_queue.cc:260] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.052001 1163 ts_tablet_manager.cc:1428] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:20.053690 1163 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.056174 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 421) took 1531 ms. Trace:
I20250625 07:52:20.058009 1163 tablet_bootstrap.cc:654] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:20.058919 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bccda893fd5466886a4d253172309e1 consensus state has no leader
I20250625 07:52:20.061749 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e972e4cb0fb94531bc712f6133830d96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.062490 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e972e4cb0fb94531bc712f6133830d96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.062835 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ee0a0386d5404189ac77d4d731677aba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.063419 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee0a0386d5404189ac77d4d731677aba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.064448 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.065059 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daf115dd3aa548dc802ed0362c274d53 has no consensus state
I20250625 07:52:20.065878 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f191436f3b9244eaa12d211b6f125175
I20250625 07:52:20.066421 594 catalog_manager.cc:6652] ReplaceTablet: tablet f191436f3b9244eaa12d211b6f125175 deleted and replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2
W20250625 07:52:20.066707 713 leader_election.cc:343] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 53c6c51a90754878977aeeaacf4773a6
I20250625 07:52:20.067026 1517 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 07:52:20.067135 745 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:20.069732 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 506794253c84486faa926de885c48c58 consensus state has no leader
W20250625 07:52:20.070815 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.071704 1543 raft_consensus.cc:491] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.071875 1543 raft_consensus.cc:513] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.072489 1543 leader_election.cc:290] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:20.073758 713 leader_election.cc:343] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 59df0f66116e4dfd9386d107cab0aaf6
I20250625 07:52:20.073956 1162 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.074119 1162 ts_tablet_manager.cc:1397] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.003s
I20250625 07:52:20.074923 1162 raft_consensus.cc:357] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.075191 1162 raft_consensus.cc:383] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.075279 1162 raft_consensus.cc:738] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.075506 1162 consensus_queue.cc:260] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.076027 1162 ts_tablet_manager.cc:1428] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:20.076400 1162 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.078081 1162 tablet_bootstrap.cc:654] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.079758 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f5b40bffec5a46e1bee5f24f77334c4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.079706 1161 rpcz_store.cc:276] 0625 07:52:18.452734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.453640 (+ 906us) service_pool.cc:225] Handling call
0625 07:52:19.966982 (+1513342us) ts_tablet_manager.cc:1145] Deleting tablet 4fdb6a689ce6440ab7037484e2b9cdc5
0625 07:52:19.966992 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:19.971332 (+ 4340us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.974572 (+ 3240us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:19.983638 (+ 9066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500712}
W20250625 07:52:20.080250 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5b40bffec5a46e1bee5f24f77334c4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.086325 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.086326 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 434) took 1541 ms. Trace:
I20250625 07:52:20.082662 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 813f9f9fccc34685b73847c9d590ceb6 (DEFAULT_TABLE table=default.Table_adb5d0acb48c45b781c468e36e59b4c0 [id=3f984673a2964bf19fdbb95a9da4e109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.087131 1161 rpcz_store.cc:276] 0625 07:52:18.541590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.542024 (+ 434us) service_pool.cc:225] Handling call
0625 07:52:20.080020 (+1537996us) ts_tablet_manager.cc:1145] Deleting tablet 9a90bb52358f4c38a540ad025e9e0430
0625 07:52:20.080030 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.083084 (+ 3054us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1523874}
I20250625 07:52:20.087424 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f9f9fccc34685b73847c9d590ceb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.087579 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 440) took 1526 ms. Trace:
I20250625 07:52:20.089982 1161 rpcz_store.cc:276] 0625 07:52:18.561806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.563078 (+ 1272us) service_pool.cc:225] Handling call
0625 07:52:20.087427 (+1524349us) ts_tablet_manager.cc:1145] Deleting tablet 547b758c2e6c4558bb232f14b37aa81a
0625 07:52:20.087437 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.087543 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492170}
I20250625 07:52:20.090446 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 443) took 1528 ms. Trace:
I20250625 07:52:20.090588 1161 rpcz_store.cc:276] 0625 07:52:18.562212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.595334 (+ 33122us) service_pool.cc:225] Handling call
0625 07:52:20.090292 (+1494958us) ts_tablet_manager.cc:1145] Deleting tablet e181a3f1ae2c49349718d842c166b1e6
0625 07:52:20.090304 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.090412 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494704}
I20250625 07:52:20.090955 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 452) took 1202 ms. Trace:
I20250625 07:52:20.091061 1161 rpcz_store.cc:276] 0625 07:52:18.888359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.923272 (+ 34913us) service_pool.cc:225] Handling call
0625 07:52:20.090828 (+1167556us) ts_tablet_manager.cc:1145] Deleting tablet 1c8694f09dcf47ab9f5fc8854c21c331
0625 07:52:20.090838 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.090922 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147708}
I20250625 07:52:20.091416 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 453) took 1202 ms. Trace:
I20250625 07:52:20.091522 1161 rpcz_store.cc:276] 0625 07:52:18.888456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:18.943154 (+ 54698us) service_pool.cc:225] Handling call
0625 07:52:20.091274 (+1148120us) ts_tablet_manager.cc:1145] Deleting tablet 9f1d251171314118b66e3a921a4a50c5
0625 07:52:20.091286 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.091382 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147934}
I20250625 07:52:20.096143 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.096151 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 4fdb6a689ce6440ab7037484e2b9cdc5 (table default.Table_3c1cbfecab414abaa74c66f1773f817e [id=5174ca174ab24d7695eb1fddc9414c74]) successfully deleted
W20250625 07:52:20.096612 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 26d2bb8b0a8f45079083f2b42bc25011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26d2bb8b0a8f45079083f2b42bc25011 already in progress: creating tablet
I20250625 07:52:20.098371 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59df0f66116e4dfd9386d107cab0aaf6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.098340 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.098388 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53c6c51a90754878977aeeaacf4773a6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.099491 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet aec412a15b964060a24631e7ebcf1ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 10c7f4b4187c428bbec136c85219da4d) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.100279 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.101039 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.101486 1161 tablet_replica.cc:331] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:20.102284 751 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.105350 1161 raft_consensus.cc:2241] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.105870 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.106077 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.106289 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7e46f27344b848318dcd1e782819b75e (DEFAULT_TABLE table=default.Table_e35c55dc42af4bb28966deeb87d520c8 [id=4fff90a74052472bb7bb8778a9786b02]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:20.102294 713 leader_election.cc:343] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.106679 713 leader_election.cc:304] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.107005 1543 raft_consensus.cc:2747] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.107087 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 76fe692067954856908f936ced332c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:20.107470 713 leader_election.cc:343] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.107627 713 leader_election.cc:304] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:20.107811 713 raft_consensus.cc:2684] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:20.108083 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.108726 1161 raft_consensus.cc:2270] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.109326 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 1f771628379842989b9b2099de65e2a0 (DEFAULT_TABLE table=default.Table_aab39ebeac1742cda147feb57cf092d7 [id=63a78517d1cb4d6790a9422cb7b822aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.109503 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2384bcb9ddc1469daf7ed5f61f44a71e" } new_table_name: "default.Table_da75eea0e9054b62ac5c228ce0fa5b27" modify_external_catalogs: true
I20250625 07:52:20.110491 1161 ts_tablet_manager.cc:1905] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.110602 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e46f27344b848318dcd1e782819b75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.111781 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f771628379842989b9b2099de65e2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.111756 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4208ed1cf6fc494f86365f0b22d2e390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.112342 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4208ed1cf6fc494f86365f0b22d2e390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:20.112304 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9a90bb52358f4c38a540ad025e9e0430 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a90bb52358f4c38a540ad025e9e0430 already in progress: creating tablet
I20250625 07:52:20.115660 1160 ts_tablet_manager.cc:1918] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:20.115677 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 547b758c2e6c4558bb232f14b37aa81a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 547b758c2e6c4558bb232f14b37aa81a already in progress: creating tablet
I20250625 07:52:20.115805 1160 log.cc:1199] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6606a9559e6b47d28669083af5aee5bd
W20250625 07:52:20.115901 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e181a3f1ae2c49349718d842c166b1e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e181a3f1ae2c49349718d842c166b1e6 already in progress: creating tablet
I20250625 07:52:20.116156 1160 ts_tablet_manager.cc:1939] T 6606a9559e6b47d28669083af5aee5bd P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:20.116422 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1c8694f09dcf47ab9f5fc8854c21c331 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c8694f09dcf47ab9f5fc8854c21c331 already in progress: creating tablet
W20250625 07:52:20.117601 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9f1d251171314118b66e3a921a4a50c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f1d251171314118b66e3a921a4a50c5 already in progress: creating tablet
W20250625 07:52:20.117887 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ea62dfb8f20f4caeaa79ea1e85d99930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62dfb8f20f4caeaa79ea1e85d99930 already in progress: creating tablet
W20250625 07:52:20.119369 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 32a9f4cf6838411fab06fa6d35ae9492 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a9f4cf6838411fab06fa6d35ae9492 already in progress: creating tablet
I20250625 07:52:20.120406 895 tablet_service.cc:1468] Processing CreateTablet for tablet 1f771628379842989b9b2099de65e2a0 (DEFAULT_TABLE table=default.Table_aab39ebeac1742cda147feb57cf092d7 [id=63a78517d1cb4d6790a9422cb7b822aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.121748 1163 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:20.122040 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 59df0f66116e4dfd9386d107cab0aaf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59df0f66116e4dfd9386d107cab0aaf6 already in progress: creating tablet
I20250625 07:52:20.121919 1163 ts_tablet_manager.cc:1397] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.003s
W20250625 07:52:20.122711 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce already in progress: creating tablet
I20250625 07:52:20.122888 1163 raft_consensus.cc:357] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:20.124070 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 871b603cb92249c6b0f16c825bf46232 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871b603cb92249c6b0f16c825bf46232 already in progress: creating tablet
I20250625 07:52:20.124670 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f771628379842989b9b2099de65e2a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.124936 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f84acd45845c4917b7dc8c5ad5232897 consensus state has no leader
I20250625 07:52:20.126940 1163 raft_consensus.cc:383] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.127048 1163 raft_consensus.cc:738] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.127305 1163 consensus_queue.cc:260] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.127880 1163 ts_tablet_manager.cc:1428] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250625 07:52:20.128337 1485 raft_consensus.cc:491] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.128495 1485 raft_consensus.cc:513] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.129012 1163 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.120265 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]). Expected version 2 got 0
I20250625 07:52:20.130205 1485 leader_election.cc:290] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:20.132509 1158 tablet_bootstrap.cc:492] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.133315 1160 tablet_replica.cc:331] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:20.134001 1160 raft_consensus.cc:2241] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.134217 1160 raft_consensus.cc:2270] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.135150 1160 ts_tablet_manager.cc:1905] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:20.135568 985 leader_election.cc:343] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:20.134629 1163 tablet_bootstrap.cc:654] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:20.136770 986 leader_election.cc:343] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:20.137688 986 leader_election.cc:304] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:20.137893 1158 ts_tablet_manager.cc:1397] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.238s user 0.051s sys 0.052s
I20250625 07:52:20.143342 1485 raft_consensus.cc:2747] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.127434 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6606a9559e6b47d28669083af5aee5bd (table default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]) successfully deleted
I20250625 07:52:20.144656 1162 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.144856 1162 ts_tablet_manager.cc:1397] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.005s
W20250625 07:52:20.144951 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 26d2bb8b0a8f45079083f2b42bc25011 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26d2bb8b0a8f45079083f2b42bc25011 already in progress: creating tablet
I20250625 07:52:20.145712 1162 raft_consensus.cc:357] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.145902 1162 raft_consensus.cc:383] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.145989 1162 raft_consensus.cc:738] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.146209 1162 consensus_queue.cc:260] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.146703 1162 ts_tablet_manager.cc:1428] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:20.147156 1162 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.149072 1162 tablet_bootstrap.cc:654] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.151036 1161 ts_tablet_manager.cc:1918] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.151345 1161 log.cc:1199] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1eab049ad7fc4b5581e5c31a14491306
W20250625 07:52:20.151883 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a1baf94a051e485eaf4c0e074e147252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1baf94a051e485eaf4c0e074e147252 already in progress: creating tablet
I20250625 07:52:20.154247 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.154978 1158 raft_consensus.cc:357] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.155170 1158 raft_consensus.cc:383] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.155908 1158 raft_consensus.cc:738] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.156168 1158 consensus_queue.cc:260] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.158715 1158 ts_tablet_manager.cc:1428] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250625 07:52:20.159422 1161 ts_tablet_manager.cc:1939] T 1eab049ad7fc4b5581e5c31a14491306 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:20.159776 1158 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:20.159548 1166 ts_tablet_manager.cc:1918] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.159993 1166 log.cc:1199] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/53c6c51a90754878977aeeaacf4773a6
I20250625 07:52:20.165027 1163 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.165203 1163 ts_tablet_manager.cc:1397] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20250625 07:52:20.165230 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 95a763abda074d358f176fd7e7be0e5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.166244 1163 raft_consensus.cc:357] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.166445 1163 raft_consensus.cc:383] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.166553 1163 raft_consensus.cc:738] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.166823 1161 tablet_replica.cc:331] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:20.166797 1163 consensus_queue.cc:260] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.167327 1163 ts_tablet_manager.cc:1428] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:20.167536 1161 raft_consensus.cc:2241] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.167640 1163 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.168172 1158 tablet_bootstrap.cc:654] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.168339 1161 raft_consensus.cc:2270] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.169883 1161 ts_tablet_manager.cc:1905] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.173672 1160 ts_tablet_manager.cc:1918] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.174292 1160 log.cc:1199] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/81976db0a2f94128a89842a12f067231
I20250625 07:52:20.174769 1160 ts_tablet_manager.cc:1939] T 81976db0a2f94128a89842a12f067231 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:20.175339 1163 tablet_bootstrap.cc:654] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.160645 1166 ts_tablet_manager.cc:1939] T 53c6c51a90754878977aeeaacf4773a6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.168741 1162 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.173271 1543 raft_consensus.cc:491] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.177191 1162 ts_tablet_manager.cc:1397] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20250625 07:52:20.177137 1543 raft_consensus.cc:513] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.177827 1543 leader_election.cc:290] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.177943 1166 tablet_replica.cc:331] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.178040 1162 raft_consensus.cc:357] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.178156 1166 raft_consensus.cc:2241] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.178221 1162 raft_consensus.cc:383] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.178289 1162 raft_consensus.cc:738] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.178314 1166 raft_consensus.cc:2270] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.178417 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:20.178507 1162 consensus_queue.cc:260] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.178608 914 raft_consensus.cc:2466] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:20.178845 1166 ts_tablet_manager.cc:1905] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.178980 1162 ts_tablet_manager.cc:1428] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:20.179791 1162 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.181694 1162 tablet_bootstrap.cc:654] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.182799 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 1eab049ad7fc4b5581e5c31a14491306 (table default.Table_3e87722f6f3e42048774f5c2a33ff506 [id=61c85476a3bb42b082c6a590daa2f7d2]) successfully deleted
I20250625 07:52:20.185120 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.186470 713 leader_election.cc:304] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:20.186764 1543 raft_consensus.cc:2802] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:20.186862 1543 raft_consensus.cc:491] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:20.186964 1543 raft_consensus.cc:3058] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.187348 1158 tablet_bootstrap.cc:492] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.187572 1158 ts_tablet_manager.cc:1397] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.004s
I20250625 07:52:20.188385 1158 raft_consensus.cc:357] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.184852 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 53c6c51a90754878977aeeaacf4773a6 (table default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]) successfully deleted
I20250625 07:52:20.188665 1158 raft_consensus.cc:383] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.188767 1158 raft_consensus.cc:738] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.189337 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 81976db0a2f94128a89842a12f067231 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
I20250625 07:52:20.189675 1158 consensus_queue.cc:260] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.189846 1543 raft_consensus.cc:513] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.190452 1543 leader_election.cc:290] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.190518 1158 ts_tablet_manager.cc:1428] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:20.190976 1158 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:20.191165 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:20.191362 914 raft_consensus.cc:3058] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.194285 914 raft_consensus.cc:2466] T 4914a64e704a4386b8241e95229c3786 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:20.195313 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]). Expected version 2 got 0
W20250625 07:52:20.196580 713 leader_election.cc:343] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.196621 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:20.197214 713 leader_election.cc:304] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:20.197563 713 leader_election.cc:343] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.198683 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]). Expected version 2 got 0
I20250625 07:52:20.199214 1162 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.197750 1543 raft_consensus.cc:2802] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:20.200400 1543 raft_consensus.cc:695] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:20.200726 1543 consensus_queue.cc:237] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:20.201855 1162 ts_tablet_manager.cc:1397] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20250625 07:52:20.202018 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0f50983dec264e34b2ed511cd2925785" } modify_external_catalogs: true
I20250625 07:52:20.202232 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0f50983dec264e34b2ed511cd2925785" } modify_external_catalogs: true
I20250625 07:52:20.202792 1162 raft_consensus.cc:357] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.203043 1162 raft_consensus.cc:383] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.203102 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9caecbdf269849638e585aac6422de32" } new_table_name: "default.Table_35586b691c9a48068784a8147f80b6ef" modify_external_catalogs: true
I20250625 07:52:20.203161 1162 raft_consensus.cc:738] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.203137 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_83f5d615c53042dd888e74c415cc3caf" } modify_external_catalogs: true
I20250625 07:52:20.203349 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_83f5d615c53042dd888e74c415cc3caf" } modify_external_catalogs: true
I20250625 07:52:20.203416 1162 consensus_queue.cc:260] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.203963 1162 ts_tablet_manager.cc:1428] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:20.204039 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_efc0d4ed62e24872b76dc943522353cd" } modify_external_catalogs: true
I20250625 07:52:20.204198 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_efc0d4ed62e24872b76dc943522353cd" } modify_external_catalogs: true
I20250625 07:52:20.204437 1162 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.204496 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_29b24dec640147299371e6748d35cc5b" } modify_external_catalogs: true
I20250625 07:52:20.204679 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6e3be4354c248d8abb373c9f0b5c6c9" } modify_external_catalogs: true
I20250625 07:52:20.204885 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6e3be4354c248d8abb373c9f0b5c6c9" } modify_external_catalogs: true
I20250625 07:52:20.204882 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_29b24dec640147299371e6748d35cc5b" } modify_external_catalogs: true
I20250625 07:52:20.208364 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c363caab5c8c4e61b584674a8c476ac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.208948 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c363caab5c8c4e61b584674a8c476ac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.214795 1162 tablet_bootstrap.cc:654] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:20.224970 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 506794253c84486faa926de885c48c58 consensus state has no leader
I20250625 07:52:20.228637 1158 tablet_bootstrap.cc:654] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.229256 1161 ts_tablet_manager.cc:1918] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.237280 1161 log.cc:1199] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6b1eaf0266c04873b185c1c0c2da201c
I20250625 07:52:20.238669 1161 ts_tablet_manager.cc:1939] T 6b1eaf0266c04873b185c1c0c2da201c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:20.239662 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6b1eaf0266c04873b185c1c0c2da201c (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
W20250625 07:52:20.240002 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3bed3f04c60249aaa389c26788974f58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed3f04c60249aaa389c26788974f58 already in progress: creating tablet
I20250625 07:52:20.240088 1161 tablet_replica.cc:331] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:20.240271 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5179a144a9a0445f933ca02e79bbc122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5179a144a9a0445f933ca02e79bbc122 already in progress: creating tablet
I20250625 07:52:20.240393 1161 raft_consensus.cc:2241] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.240680 1161 raft_consensus.cc:2270] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.241593 1161 ts_tablet_manager.cc:1905] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.221870 1166 ts_tablet_manager.cc:1918] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.245774 1166 log.cc:1199] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/59df0f66116e4dfd9386d107cab0aaf6
I20250625 07:52:20.246219 1166 ts_tablet_manager.cc:1939] T 59df0f66116e4dfd9386d107cab0aaf6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.202879 1160 tablet_replica.cc:331] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:20.247718 1160 raft_consensus.cc:2241] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.247937 1160 raft_consensus.cc:2270] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.253708 1166 tablet_replica.cc:331] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.254820 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 59df0f66116e4dfd9386d107cab0aaf6 (table default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]) successfully deleted
I20250625 07:52:20.256371 1166 raft_consensus.cc:2241] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.258587 1158 tablet_bootstrap.cc:492] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.258878 1158 ts_tablet_manager.cc:1397] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250625 07:52:20.258956 594 master_service.cc:948] ReplaceTablet: received request to replace tablet f4bb74391a1c4e2792226d259a07f977 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:20.258989 1166 raft_consensus.cc:2270] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.257844 1160 ts_tablet_manager.cc:1905] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.259781 1166 ts_tablet_manager.cc:1905] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.259989 1158 raft_consensus.cc:357] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.260236 1158 raft_consensus.cc:383] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.260406 1158 raft_consensus.cc:738] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.260767 1158 consensus_queue.cc:260] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.261526 1158 ts_tablet_manager.cc:1428] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:20.261897 1158 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:20.268210 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 32bd7cdef96946e88683e457c2f26c9b with delete_type TABLET_DATA_DELETED (replaced by tablet 2db37ece7a964f47911fe4fc7434cd07) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.271168 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_feabe5a9dd014d698a57f45b19bad2cb" } new_table_name: "default.Table_705c8d46163b48cc9cf55cdf1aa9e887" modify_external_catalogs: true
I20250625 07:52:20.272668 1158 tablet_bootstrap.cc:654] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.273351 1162 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.273576 1162 ts_tablet_manager.cc:1397] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.003s
I20250625 07:52:20.274435 1162 raft_consensus.cc:357] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.274698 1162 raft_consensus.cc:383] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.274824 1162 raft_consensus.cc:738] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.276562 1162 consensus_queue.cc:260] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.277201 1162 ts_tablet_manager.cc:1428] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:20.277595 1162 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:20.278436 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23908409fbbd48dbbe82be8182ba665f has no consensus state
I20250625 07:52:20.278844 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f7740a0ca4b24e5899c7fec81bb78e8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.279450 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7740a0ca4b24e5899c7fec81bb78e8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.286163 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0fb32606c55d45a7878009be37eb3ca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.286749 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fb32606c55d45a7878009be37eb3ca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.291952 1162 tablet_bootstrap.cc:654] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.295380 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0a167e3ae60a42e7a47dca60a52078f8" } new_table_name: "default.Table_1cca330896134062a3528e0f5dca94c4" modify_external_catalogs: true
I20250625 07:52:20.299178 1536 raft_consensus.cc:491] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.299389 1536 raft_consensus.cc:513] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.299468 1161 ts_tablet_manager.cc:1918] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.300065 1536 leader_election.cc:290] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.300122 1161 log.cc:1199] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/aec412a15b964060a24631e7ebcf1ef7
I20250625 07:52:20.300536 1161 ts_tablet_manager.cc:1939] T aec412a15b964060a24631e7ebcf1ef7 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:20.300647 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29eaa8d93e5b4d1d8ed147b60c7da859" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.301344 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29eaa8d93e5b4d1d8ed147b60c7da859" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:20.302282 851 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:20.302677 850 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.302789 850 leader_election.cc:304] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.303056 1536 raft_consensus.cc:2747] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.303260 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet aec412a15b964060a24631e7ebcf1ef7 (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:20.304080 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e66f393d52d48938a989a0c7104879c" } new_table_name: "default.Table_e01d3c58925248249dc9bff2b7be111c" modify_external_catalogs: true
W20250625 07:52:20.304818 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2082e5e28c5c4d1fac7e40acf77f75e6 consensus state has no leader
I20250625 07:52:20.307111 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b43beebf15d44d2a37382c52d1c35fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.307476 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet faf049714a7e4d7eb947c29f272bbb7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faf049714a7e4d7eb947c29f272bbb7c already in progress: creating tablet
W20250625 07:52:20.307858 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b43beebf15d44d2a37382c52d1c35fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:20.308947 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1898ff5b7626478bb997f7a433477498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1898ff5b7626478bb997f7a433477498 already in progress: creating tablet
W20250625 07:52:20.310671 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62aaff3c000e4e69b52222aa47260c3a consensus state has no leader
I20250625 07:52:20.311106 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e53a5a947f2c4463a268d7c905ee326f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.311367 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f191436f3b9244eaa12d211b6f125175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f191436f3b9244eaa12d211b6f125175 already in progress: creating tablet
I20250625 07:52:20.313282 1333 raft_consensus.cc:491] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:20.313048 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e53a5a947f2c4463a268d7c905ee326f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.313454 1333 raft_consensus.cc:513] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:20.313051 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a1f823fd3b47c9b6d1cf5647f9a68a already in progress: creating tablet
I20250625 07:52:20.314138 1333 leader_election.cc:290] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.315249 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.315557 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:20.315889 914 raft_consensus.cc:2466] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:20.316293 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 76fe692067954856908f936ced332c18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76fe692067954856908f936ced332c18 already in progress: creating tablet
I20250625 07:52:20.317452 1048 raft_consensus.cc:3058] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.318667 1161 tablet_replica.cc:331] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:20.320297 1163 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.320551 1163 ts_tablet_manager.cc:1397] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.153s user 0.002s sys 0.002s
I20250625 07:52:20.321388 1163 raft_consensus.cc:357] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.322121 1163 raft_consensus.cc:383] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.322258 1163 raft_consensus.cc:738] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.323290 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c24353dd31c0489299840521478b82f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.324537 1163 consensus_queue.cc:260] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.326239 1163 ts_tablet_manager.cc:1428] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:20.328136 1163 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.330744 1163 tablet_bootstrap.cc:654] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.339836 1161 raft_consensus.cc:2241] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.336169 1536 raft_consensus.cc:491] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.340843 1536 raft_consensus.cc:513] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.341568 1536 leader_election.cc:290] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.342716 713 leader_election.cc:304] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:20.343087 713 leader_election.cc:343] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.343832 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:20.344287 1543 raft_consensus.cc:2802] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:20.344385 1543 raft_consensus.cc:491] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:20.344507 1543 raft_consensus.cc:3058] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.344895 1161 raft_consensus.cc:2270] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.346799 1162 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.346974 1162 ts_tablet_manager.cc:1397] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.004s
I20250625 07:52:20.346850 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f442ff326d064398b181ffcb29aa6439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.347447 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f442ff326d064398b181ffcb29aa6439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.347506 1543 raft_consensus.cc:513] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.347818 1162 raft_consensus.cc:357] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.348016 1162 raft_consensus.cc:383] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:20.348904 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c24353dd31c0489299840521478b82f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:20.349812 851 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.351799 1543 leader_election.cc:290] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.352756 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:20.352866 1162 raft_consensus.cc:738] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.352981 914 raft_consensus.cc:3058] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.353142 1162 consensus_queue.cc:260] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.353646 1162 ts_tablet_manager.cc:1428] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250625 07:52:20.353989 1162 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.355628 1162 tablet_bootstrap.cc:654] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.355952 914 raft_consensus.cc:2466] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:20.358950 1161 ts_tablet_manager.cc:1905] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.360318 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.364091 850 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.364243 850 leader_election.cc:304] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.364800 1536 raft_consensus.cc:2747] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.364976 1536 raft_consensus.cc:491] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.365125 1536 raft_consensus.cc:513] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.365705 1536 leader_election.cc:290] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.366612 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:20.367146 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e928120d4b924a9586fa93e2c9fd97bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.367322 851 leader_election.cc:343] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:20.367738 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e928120d4b924a9586fa93e2c9fd97bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.372092 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.372754 850 leader_election.cc:343] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.372913 850 leader_election.cc:304] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.373209 1158 tablet_bootstrap.cc:492] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.373261 1536 raft_consensus.cc:2747] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.373889 713 leader_election.cc:304] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:20.374168 1543 raft_consensus.cc:2802] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:20.374406 1158 ts_tablet_manager.cc:1397] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.006s
I20250625 07:52:20.374801 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa30ca0035048d3b60669994d402734" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:20.375272 1543 raft_consensus.cc:695] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:20.375608 1158 raft_consensus.cc:357] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.375921 1158 raft_consensus.cc:383] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.375732 1543 consensus_queue.cc:237] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.376009 1158 raft_consensus.cc:738] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.376345 1158 consensus_queue.cc:260] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.376965 1158 ts_tablet_manager.cc:1428] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:20.377394 1158 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:20.377629 713 leader_election.cc:343] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.380266 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d25d399f7faa4e56a98aeee2783040e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.385432 1160 ts_tablet_manager.cc:1918] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.388404 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1973678168a84875b747897f08dd1bad
I20250625 07:52:20.389381 1158 tablet_bootstrap.cc:654] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.389415 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6846d6c9101b41eba83b31905770dd57" } new_table_name: "default.Table_9650a59cdcb04bf395e5cc36090a99f1" modify_external_catalogs: true
I20250625 07:52:20.389493 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_43525db1d2034f6f8460bbfccb691fc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.391309 1161 ts_tablet_manager.cc:1918] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.391431 1161 log.cc:1199] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/95a763abda074d358f176fd7e7be0e5c
W20250625 07:52:20.391772 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43525db1d2034f6f8460bbfccb691fc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.391856 1161 ts_tablet_manager.cc:1939] T 95a763abda074d358f176fd7e7be0e5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:20.392057 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 1973678168a84875b747897f08dd1bad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.392330 1161 tablet_replica.cc:331] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:20.392526 1161 raft_consensus.cc:2241] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.392545 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 95a763abda074d358f176fd7e7be0e5c (table default.Table_1e843117005043efadef968ca11c482f [id=44ba0323288e45269c5dda7cd7f60f84]) successfully deleted
I20250625 07:52:20.393584 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 1973678168a84875b747897f08dd1bad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.396752 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 0b7daf759e324c1db853fdbaab76c721 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:20.401150 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]). Expected version 2 got 0
I20250625 07:52:20.401417 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_c632d0f66e464da088ba6a64e1d70e3b [id=5d0b8b49ed7f45aeae69c23c09582b34]). Expected version 2 got 0
I20250625 07:52:20.403604 1161 raft_consensus.cc:2270] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:20.407500 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d25d399f7faa4e56a98aeee2783040e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.409464 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:20.410238 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9578d77db00242d48f4e8f6f8263ef09
I20250625 07:52:20.411181 1166 ts_tablet_manager.cc:1918] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:20.413656 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f655669b74b4a6594fbd63c45d15bf4 consensus state has no leader
I20250625 07:52:20.414989 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 8bc8f47541c54c4287ebb7b892daa1be from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:20.405119 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da (DEFAULT_TABLE table=default.Table_ea01ec621c614388990c5fddddbd17a5 [id=01f430fe728f4053932bcabfa1c9b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.415978 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.417456 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 97b9a14151fc472e956126620633be32
I20250625 07:52:20.420164 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.412441 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.412619 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.425331 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:20.426539 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.426756 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ac06645d7b71464ab4e4e92402c14a13" } new_table_name: "default.Table_457927be3bfb45d1ab90fc5a3b8163d8" modify_external_catalogs: true
I20250625 07:52:20.414176 746 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da (DEFAULT_TABLE table=default.Table_ea01ec621c614388990c5fddddbd17a5 [id=01f430fe728f4053932bcabfa1c9b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.427950 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.428020 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.414805 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.414873 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.429040 1161 ts_tablet_manager.cc:1905] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.429945 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9eaab8f86bdc45cab8645cd5a6e3489c (DEFAULT_TABLE table=default.Table_f9279117557549b3826d8056f0d22581 [id=54e5a28725fb48609e29a7a01c76a3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.430351 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eaab8f86bdc45cab8645cd5a6e3489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.430546 1158 tablet_bootstrap.cc:492] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.430681 1158 ts_tablet_manager.cc:1397] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.002s
I20250625 07:52:20.430436 592 catalog_manager.cc:5582] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } }
I20250625 07:52:20.432653 592 catalog_manager.cc:5582] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:20.434419 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a37bdd5dd7c1405ea495ec1a66f47133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.427516 1162 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.434959 1162 ts_tablet_manager.cc:1397] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.001s
W20250625 07:52:20.434983 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a37bdd5dd7c1405ea495ec1a66f47133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.435707 1162 raft_consensus.cc:357] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.435925 1162 raft_consensus.cc:383] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.436003 1162 raft_consensus.cc:738] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.436239 1162 consensus_queue.cc:260] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.436841 1162 ts_tablet_manager.cc:1428] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:20.436962 752 tablet_service.cc:1468] Processing CreateTablet for tablet 9eaab8f86bdc45cab8645cd5a6e3489c (DEFAULT_TABLE table=default.Table_f9279117557549b3826d8056f0d22581 [id=54e5a28725fb48609e29a7a01c76a3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.437135 746 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.437402 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eaab8f86bdc45cab8645cd5a6e3489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.437481 1162 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.439323 1162 tablet_bootstrap.cc:654] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.440271 1166 log.cc:1199] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e76df9405d91488dad0cfa7f8755bb1c
I20250625 07:52:20.440665 1166 ts_tablet_manager.cc:1939] T e76df9405d91488dad0cfa7f8755bb1c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.441975 1166 tablet_replica.cc:331] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.442236 1166 raft_consensus.cc:2241] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.442440 1166 raft_consensus.cc:2270] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.443636 1166 ts_tablet_manager.cc:1905] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.446379 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f4bb74391a1c4e2792226d259a07f977
I20250625 07:52:20.446872 594 catalog_manager.cc:6652] ReplaceTablet: tablet f4bb74391a1c4e2792226d259a07f977 deleted and replaced by tablet dfc22a46594f47deb8b142bc14c81a0b
I20250625 07:52:20.451258 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e76df9405d91488dad0cfa7f8755bb1c (table default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]) successfully deleted
W20250625 07:52:20.451779 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f191436f3b9244eaa12d211b6f125175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f191436f3b9244eaa12d211b6f125175 already in progress: creating tablet
I20250625 07:52:20.452092 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 57c3d0810cf44a33b2acd50707931d26 (DEFAULT_TABLE table=default.Table_e0319c343efd4139bb3d83e99569aa6c [id=e54da0da99f744839ff8cf175e23efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.452581 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c3d0810cf44a33b2acd50707931d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.452623 1536 raft_consensus.cc:491] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.454196 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.454414 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.454605 883 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.454725 752 tablet_service.cc:1468] Processing CreateTablet for tablet 57c3d0810cf44a33b2acd50707931d26 (DEFAULT_TABLE table=default.Table_e0319c343efd4139bb3d83e99569aa6c [id=e54da0da99f744839ff8cf175e23efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.455164 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c3d0810cf44a33b2acd50707931d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.455111 894 tablet_service.cc:1468] Processing CreateTablet for tablet 57c3d0810cf44a33b2acd50707931d26 (DEFAULT_TABLE table=default.Table_e0319c343efd4139bb3d83e99569aa6c [id=e54da0da99f744839ff8cf175e23efda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.455693 892 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 (DEFAULT_TABLE table=default.Table_b36659e6e06d41298f3a614061f97b73 [id=76b5f05d74fa415a93eb6df1c6da6a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.458293 746 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 (DEFAULT_TABLE table=default.Table_b36659e6e06d41298f3a614061f97b73 [id=76b5f05d74fa415a93eb6df1c6da6a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.458668 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c150fb54c405c8dc61ac133d98b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.461076 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.462021 1536 raft_consensus.cc:513] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.462455 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 47b4112b7e674518bcc1ea748370fd85 (DEFAULT_TABLE table=default.Table_f8c817fc80f64e76a3078837e90e97ce [id=5f7f3e323b6746dd93b84da3e9fb4bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.462936 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b4112b7e674518bcc1ea748370fd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.461979 1160 log.cc:1199] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/de9ccb0a2d3846228acbc2769786a059
I20250625 07:52:20.463021 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 1973678168a84875b747897f08dd1bad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.464064 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:20.464740 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 (DEFAULT_TABLE table=default.Table_b36659e6e06d41298f3a614061f97b73 [id=76b5f05d74fa415a93eb6df1c6da6a8e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:20.464629 851 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.465436 886 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da (DEFAULT_TABLE table=default.Table_ea01ec621c614388990c5fddddbd17a5 [id=01f430fe728f4053932bcabfa1c9b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.465669 884 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.466078 1158 raft_consensus.cc:357] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.466917 1158 raft_consensus.cc:383] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.467017 1011 tablet_service.cc:1468] Processing CreateTablet for tablet b782414a682246feb2811b4e3bf8b5a2 (DEFAULT_TABLE table=default.Table_1860ecd4dd0340c39868730e3b641d4d [id=c510c526b05f4186987fa157b6827db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.467574 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c150fb54c405c8dc61ac133d98b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.468303 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.468850 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b782414a682246feb2811b4e3bf8b5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.468937 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.472254 744 tablet_service.cc:1468] Processing CreateTablet for tablet b782414a682246feb2811b4e3bf8b5a2 (DEFAULT_TABLE table=default.Table_1860ecd4dd0340c39868730e3b641d4d [id=c510c526b05f4186987fa157b6827db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.475080 1158 raft_consensus.cc:738] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.475404 1158 consensus_queue.cc:260] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.478561 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 (DEFAULT_TABLE table=default.Table_d50dc89e611a4cd490d628b4a824a34f [id=91a82ca9293d48848e491b93f04d12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.478974 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f3b7f0a904656af418d01ed6f58b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.481105 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2a5dd2a80df94b488a9c2ed687499a4e" } new_table_name: "default.Table_82cd3da25b72491089bf529b94aafabd" modify_external_catalogs: true
I20250625 07:52:20.484172 594 master_service.cc:948] ReplaceTablet: received request to replace tablet b1422987deca41a79f8db666d6961643 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:20.486366 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:20.486500 1158 ts_tablet_manager.cc:1428] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.056s user 0.002s sys 0.002s
I20250625 07:52:20.486824 1158 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:20.486899 591 catalog_manager.cc:6652] ReplaceTablet: tablet 0b7daf759e324c1db853fdbaab76c721 deleted and replaced by tablet 03d999941e2a4265a64df7ec106a89a7
I20250625 07:52:20.487329 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.490528 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b782414a682246feb2811b4e3bf8b5a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.492635 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8b5bb3fe3ef4bfdba93b862784a9f61 consensus state has no leader
I20250625 07:52:20.492894 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:20.493382 597 catalog_manager.cc:6652] ReplaceTablet: tablet 8bc8f47541c54c4287ebb7b892daa1be deleted and replaced by tablet bb03e70e9aa740f18d8cd0543f730f83
I20250625 07:52:20.495090 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:20.496428 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 62aaff3c000e4e69b52222aa47260c3a (table default.Table_e01d3c58925248249dc9bff2b7be111c [id=79d68081c3a643109237f5d5478df82c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 62aaff3c000e4e69b52222aa47260c3a consensus state has no leader
I20250625 07:52:20.496636 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2f2f17244c6943d8b807f3713710ad12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.497416 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f2f17244c6943d8b807f3713710ad12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.500984 1536 leader_election.cc:290] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.501263 1543 raft_consensus.cc:491] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.501421 1543 raft_consensus.cc:513] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.504096 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b1422987deca41a79f8db666d6961643
I20250625 07:52:20.504668 594 catalog_manager.cc:6652] ReplaceTablet: tablet b1422987deca41a79f8db666d6961643 deleted and replaced by tablet 64a5ff005b424ae5ae9831263abf58ef
I20250625 07:52:20.505556 1543 leader_election.cc:290] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.505899 751 tablet_service.cc:1468] Processing CreateTablet for tablet 47b4112b7e674518bcc1ea748370fd85 (DEFAULT_TABLE table=default.Table_f8c817fc80f64e76a3078837e90e97ce [id=5f7f3e323b6746dd93b84da3e9fb4bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.506007 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9eaab8f86bdc45cab8645cd5a6e3489c (DEFAULT_TABLE table=default.Table_f9279117557549b3826d8056f0d22581 [id=54e5a28725fb48609e29a7a01c76a3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.506444 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57c3d0810cf44a33b2acd50707931d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.506490 881 tablet_service.cc:1468] Processing CreateTablet for tablet b782414a682246feb2811b4e3bf8b5a2 (DEFAULT_TABLE table=default.Table_1860ecd4dd0340c39868730e3b641d4d [id=c510c526b05f4186987fa157b6827db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.506848 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b782414a682246feb2811b4e3bf8b5a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.507129 850 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.507263 850 leader_election.cc:304] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.508131 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.508426 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.508914 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b4112b7e674518bcc1ea748370fd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.508934 889 tablet_service.cc:1468] Processing CreateTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 (DEFAULT_TABLE table=default.Table_d50dc89e611a4cd490d628b4a824a34f [id=91a82ca9293d48848e491b93f04d12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.509271 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f3b7f0a904656af418d01ed6f58b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.511382 884 tablet_service.cc:1468] Processing CreateTablet for tablet 47b4112b7e674518bcc1ea748370fd85 (DEFAULT_TABLE table=default.Table_f8c817fc80f64e76a3078837e90e97ce [id=5f7f3e323b6746dd93b84da3e9fb4bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.511729 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b4112b7e674518bcc1ea748370fd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.513082 895 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.515340 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9eaab8f86bdc45cab8645cd5a6e3489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.516945 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1c150fb54c405c8dc61ac133d98b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.517153 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bd784c92cfd3446e8177e7bac2fa84fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.517722 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd784c92cfd3446e8177e7bac2fa84fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.519387 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_595ab735818843488af81e560f201d30" } new_table_name: "default.Table_a9c6f935691440fe82ca615932a43240" modify_external_catalogs: true
I20250625 07:52:20.519443 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0ba3665554d4b36a261ce71b5fd743c" } new_table_name: "default.Table_1225d003be2b4d048266736844a78a01" modify_external_catalogs: true
W20250625 07:52:20.524119 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7c6d068d2324ea6a07bb9be74578870 consensus state has no leader
I20250625 07:52:20.525434 1162 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.525585 1162 ts_tablet_manager.cc:1397] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.000s
I20250625 07:52:20.464776 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.526504 1162 raft_consensus.cc:357] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.526686 1162 raft_consensus.cc:383] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.526773 1162 raft_consensus.cc:738] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.526986 1162 consensus_queue.cc:260] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.528678 1162 ts_tablet_manager.cc:1428] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:20.529006 1162 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.530758 1162 tablet_bootstrap.cc:654] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.531693 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 29f2997d1d714100950383e6af04f315 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:20.532756 1160 ts_tablet_manager.cc:1939] T de9ccb0a2d3846228acbc2769786a059 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:20.534053 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2d792b2565234096a0bc44a9c82abc1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.534775 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d792b2565234096a0bc44a9c82abc1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.536509 895 tablet_service.cc:1468] Processing CreateTablet for tablet 9336068493fe4245b4365f5acfdacf77 (DEFAULT_TABLE table=default.Table_06acb0609d5e417886a27bc6693624ff [id=44e00612b764458ea3f6ba2c87c2a446]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.537274 893 tablet_service.cc:1468] Processing CreateTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.534723 1166 ts_tablet_manager.cc:1918] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.537485 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.537707 1536 raft_consensus.cc:491] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.537851 1166 log.cc:1199] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c6031b9695cd4a99bd0e5391663170ff
I20250625 07:52:20.537866 1536 raft_consensus.cc:513] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.538192 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.538281 1166 ts_tablet_manager.cc:1939] T c6031b9695cd4a99bd0e5391663170ff P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.538883 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c28a064006145eea000e74da10d0794" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:20.539239 1166 tablet_replica.cc:331] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.539422 1166 raft_consensus.cc:2241] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.539527 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.539669 1166 raft_consensus.cc:2270] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.539995 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b4fc14df40f3bb9a6e77699981ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.539909 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
I20250625 07:52:20.540233 1166 ts_tablet_manager.cc:1905] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.540246 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 9336068493fe4245b4365f5acfdacf77 (DEFAULT_TABLE table=default.Table_06acb0609d5e417886a27bc6693624ff [id=44e00612b764458ea3f6ba2c87c2a446]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.541013 1536 leader_election.cc:290] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.541092 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9336068493fe4245b4365f5acfdacf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.541265 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf9e6ee9a3c4dcd820e33ec7bcde1da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.541642 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.541895 1158 tablet_bootstrap.cc:654] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.542428 744 tablet_service.cc:1468] Processing CreateTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 (DEFAULT_TABLE table=default.Table_d50dc89e611a4cd490d628b4a824a34f [id=91a82ca9293d48848e491b93f04d12e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.542853 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670f3b7f0a904656af418d01ed6f58b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.543740 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9336068493fe4245b4365f5acfdacf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.545292 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c28a064006145eea000e74da10d0794" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.545708 1582 raft_consensus.cc:2747] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.546061 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.546334 1029 tablet_service.cc:1468] Processing CreateTablet for tablet daf115dd3aa548dc802ed0362c274d53 (DEFAULT_TABLE table=default.Table_e6092e518ceb4af7a5d84435008334ba [id=a5338f523e69497cb7c17df7feda72f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.546761 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf115dd3aa548dc802ed0362c274d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.547369 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.547770 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b4fc14df40f3bb9a6e77699981ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.550292 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_696ed031c49440bc8b00f3c81a7d332a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.550946 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_696ed031c49440bc8b00f3c81a7d332a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.552230 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 23908409fbbd48dbbe82be8182ba665f (DEFAULT_TABLE table=default.Table_705c8d46163b48cc9cf55cdf1aa9e887 [id=f3d401f6b64540f9a0594205585b22ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.552891 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23908409fbbd48dbbe82be8182ba665f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.553356 757 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.553982 757 tablet_service.cc:1468] Processing CreateTablet for tablet 9336068493fe4245b4365f5acfdacf77 (DEFAULT_TABLE table=default.Table_06acb0609d5e417886a27bc6693624ff [id=44e00612b764458ea3f6ba2c87c2a446]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.554477 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9336068493fe4245b4365f5acfdacf77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.555984 851 leader_election.cc:343] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.563145 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6b4bca32de4a538071a652409e93b2 (DEFAULT_TABLE table=default.Table_5cbcfdbba5a84de1ac582e4687f347a5 [id=cd272991373c4fee95c14b59535eaa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.564824 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 76fe692067954856908f936ced332c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.565243 1543 raft_consensus.cc:491] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.565434 1543 raft_consensus.cc:513] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.566056 1543 leader_election.cc:290] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.568193 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c6031b9695cd4a99bd0e5391663170ff (table default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]) successfully deleted
I20250625 07:52:20.568629 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c40382d89af6427eaf3c73a32b21d835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.569082 757 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6b4bca32de4a538071a652409e93b2 (DEFAULT_TABLE table=default.Table_5cbcfdbba5a84de1ac582e4687f347a5 [id=cd272991373c4fee95c14b59535eaa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.569154 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3758b4fc14df40f3bb9a6e77699981ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.569224 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c40382d89af6427eaf3c73a32b21d835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.569489 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6b4bca32de4a538071a652409e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.570278 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.570914 850 leader_election.cc:343] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.571161 850 leader_election.cc:304] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.572302 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6b4bca32de4a538071a652409e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.572582 1163 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.573081 754 tablet_service.cc:1468] Processing CreateTablet for tablet 23908409fbbd48dbbe82be8182ba665f (DEFAULT_TABLE table=default.Table_705c8d46163b48cc9cf55cdf1aa9e887 [id=f3d401f6b64540f9a0594205585b22ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.573338 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 962633b0d9644c308f0b7fe1db714daf (DEFAULT_TABLE table=default.Table_5fbde73edcc941ebb33753389aea2dc8 [id=a01d744167054a7bb92fd9a2eee73d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.573495 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23908409fbbd48dbbe82be8182ba665f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.574332 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962633b0d9644c308f0b7fe1db714daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.575126 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:20.576557 1166 ts_tablet_manager.cc:1918] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.580966 1166 log.cc:1199] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4384bc544af149338514d40adf683f34
I20250625 07:52:20.582021 1166 ts_tablet_manager.cc:1939] T 4384bc544af149338514d40adf683f34 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.581944 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.582538 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.583230 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a7c6d068d2324ea6a07bb9be74578870 (DEFAULT_TABLE table=default.Table_82cd3da25b72491089bf529b94aafabd [id=3ff721dec5f145209210d0ae74e38b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.583588 1166 tablet_replica.cc:331] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:20.576992 751 tablet_service.cc:1468] Processing CreateTablet for tablet daf115dd3aa548dc802ed0362c274d53 (DEFAULT_TABLE table=default.Table_e6092e518ceb4af7a5d84435008334ba [id=a5338f523e69497cb7c17df7feda72f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.583794 1166 raft_consensus.cc:2241] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.583966 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:20.584206 914 raft_consensus.cc:2466] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:20.585870 1582 raft_consensus.cc:2747] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.586395 1163 ts_tablet_manager.cc:1397] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.245s user 0.011s sys 0.005s
I20250625 07:52:20.586854 883 tablet_service.cc:1468] Processing CreateTablet for tablet 23908409fbbd48dbbe82be8182ba665f (DEFAULT_TABLE table=default.Table_705c8d46163b48cc9cf55cdf1aa9e887 [id=f3d401f6b64540f9a0594205585b22ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.584141 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf115dd3aa548dc802ed0362c274d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.587348 1160 tablet_replica.cc:331] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:20.587498 1166 raft_consensus.cc:2270] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.587566 1160 raft_consensus.cc:2241] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.587534 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6b4bca32de4a538071a652409e93b2 (DEFAULT_TABLE table=default.Table_5cbcfdbba5a84de1ac582e4687f347a5 [id=cd272991373c4fee95c14b59535eaa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.587754 1160 raft_consensus.cc:2270] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.588572 1160 ts_tablet_manager.cc:1905] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.588711 882 tablet_service.cc:1468] Processing CreateTablet for tablet 962633b0d9644c308f0b7fe1db714daf (DEFAULT_TABLE table=default.Table_5fbde73edcc941ebb33753389aea2dc8 [id=a01d744167054a7bb92fd9a2eee73d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.589318 1012 tablet_service.cc:1468] Processing CreateTablet for tablet ec1e58fe6d72451682bd47d7238560b6 (DEFAULT_TABLE table=default.Table_3de45b2a2d3049d788bc30869bd75d2f [id=8b6b6ad8052743f7972cb279120fbcdf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:20.590276 713 leader_election.cc:343] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.590349 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet de9ccb0a2d3846228acbc2769786a059 (table default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]) successfully deleted
I20250625 07:52:20.590498 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6d068d2324ea6a07bb9be74578870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.590821 880 tablet_service.cc:1468] Processing CreateTablet for tablet ec1e58fe6d72451682bd47d7238560b6 (DEFAULT_TABLE table=default.Table_3de45b2a2d3049d788bc30869bd75d2f [id=8b6b6ad8052743f7972cb279120fbcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.591379 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4384bc544af149338514d40adf683f34 (table default.Table_ddae42a67be34593a367bf6f3d9e8cde [id=fd174ee9d81e42c19cab3c058584ea20]) successfully deleted
I20250625 07:52:20.593828 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_132ff7dc86c64138af1deb2266c2accc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:20.594520 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_132ff7dc86c64138af1deb2266c2accc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.597433 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e928120d4b924a9586fa93e2c9fd97bd" } modify_external_catalogs: true
I20250625 07:52:20.597658 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e928120d4b924a9586fa93e2c9fd97bd" } modify_external_catalogs: true
I20250625 07:52:20.598489 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d92962b9e7d14c8d88d775b98e75de35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.598913 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23908409fbbd48dbbe82be8182ba665f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.600666 890 tablet_service.cc:1468] Processing CreateTablet for tablet a7c6d068d2324ea6a07bb9be74578870 (DEFAULT_TABLE table=default.Table_82cd3da25b72491089bf529b94aafabd [id=3ff721dec5f145209210d0ae74e38b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.603945 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962633b0d9644c308f0b7fe1db714daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.609200 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6b4bca32de4a538071a652409e93b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.610126 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1e58fe6d72451682bd47d7238560b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.610359 1162 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.610603 1162 ts_tablet_manager.cc:1397] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20250625 07:52:20.611434 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6d068d2324ea6a07bb9be74578870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.612782 1162 raft_consensus.cc:357] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.613029 1162 raft_consensus.cc:383] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.613108 1162 raft_consensus.cc:738] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.613345 1162 consensus_queue.cc:260] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.613852 1162 ts_tablet_manager.cc:1428] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:20.614200 1162 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.616397 1166 ts_tablet_manager.cc:1905] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.618860 1161 ts_tablet_manager.cc:1918] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.619571 1161 log.cc:1199] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/32bd7cdef96946e88683e457c2f26c9b
I20250625 07:52:20.617461 713 leader_election.cc:304] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.619951 1161 ts_tablet_manager.cc:1939] T 32bd7cdef96946e88683e457c2f26c9b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:20.620003 1543 raft_consensus.cc:2802] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:20.620098 1543 raft_consensus.cc:491] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:20.620206 1543 raft_consensus.cc:3058] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.622349 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1e58fe6d72451682bd47d7238560b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.622978 1543 raft_consensus.cc:513] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.623661 1543 leader_election.cc:290] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.624788 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 0cdee1535e5a4a21804e28e78aaea0eb (DEFAULT_TABLE table=default.Table_85849694e69e4737a9793e7c46e40a87 [id=fb8226e9c0a440629bae51d5425f6f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.625262 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdee1535e5a4a21804e28e78aaea0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.628147 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:20.629734 713 leader_election.cc:343] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:20.630023 713 leader_election.cc:343] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:20.630252 713 leader_election.cc:343] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d92962b9e7d14c8d88d775b98e75de35
I20250625 07:52:20.630381 713 leader_election.cc:304] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:20.630628 713 raft_consensus.cc:2684] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:20.632071 752 tablet_service.cc:1468] Processing CreateTablet for tablet ec1e58fe6d72451682bd47d7238560b6 (DEFAULT_TABLE table=default.Table_3de45b2a2d3049d788bc30869bd75d2f [id=8b6b6ad8052743f7972cb279120fbcdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.632380 1163 raft_consensus.cc:357] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.632589 1163 raft_consensus.cc:383] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.632634 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1e58fe6d72451682bd47d7238560b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.639674 889 tablet_service.cc:1468] Processing CreateTablet for tablet daf115dd3aa548dc802ed0362c274d53 (DEFAULT_TABLE table=default.Table_e6092e518ceb4af7a5d84435008334ba [id=a5338f523e69497cb7c17df7feda72f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.648403 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:20.651836 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_322b342d0f384574a0239ed582647def" } modify_external_catalogs: true
I20250625 07:52:20.652143 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_322b342d0f384574a0239ed582647def" } modify_external_catalogs: true
I20250625 07:52:20.640851 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:20.656482 914 raft_consensus.cc:3058] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:20.660342 914 raft_consensus.cc:2466] T 9cb2aa6dac1f4443be9782aac8f855af P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:20.660871 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf115dd3aa548dc802ed0362c274d53. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.660921 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0aeb7077591f466eb77e3096f6b1d74b consensus state has no leader
W20250625 07:52:20.661543 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fdc3220d53142faae41fd1f1500de56 consensus state has no leader
I20250625 07:52:20.670806 1163 raft_consensus.cc:738] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.670953 1162 tablet_bootstrap.cc:654] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.675647 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 32bd7cdef96946e88683e457c2f26c9b (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]) successfully deleted
I20250625 07:52:20.676100 1163 consensus_queue.cc:260] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:20.676906 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1973678168a84875b747897f08dd1bad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1973678168a84875b747897f08dd1bad already in progress: creating tablet
W20250625 07:52:20.677917 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 97b9a14151fc472e956126620633be32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b9a14151fc472e956126620633be32 already in progress: creating tablet
I20250625 07:52:20.679809 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b674a39d7f9f44708b1369a5affecc82" } modify_external_catalogs: true
I20250625 07:52:20.680011 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b674a39d7f9f44708b1369a5affecc82" } modify_external_catalogs: true
I20250625 07:52:20.680110 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5a2522dd8ce349ec8bfa57ba2d10b114" } new_table_name: "default.Table_2c27be0b9107412facdd1bc4d768eebb" modify_external_catalogs: true
W20250625 07:52:20.680655 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7e60d4f9aed44082b9905fbb44ffb110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e60d4f9aed44082b9905fbb44ffb110 already in progress: creating tablet
I20250625 07:52:20.681030 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_14bfc8ca6e024a2bba684efdcfc28fd3" } modify_external_catalogs: true
I20250625 07:52:20.681182 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_14bfc8ca6e024a2bba684efdcfc28fd3" } modify_external_catalogs: true
W20250625 07:52:20.681506 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9578d77db00242d48f4e8f6f8263ef09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9578d77db00242d48f4e8f6f8263ef09 already in progress: creating tablet
I20250625 07:52:20.681721 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_786d6b63b6904ffe87ce1f3eb9769954" } modify_external_catalogs: true
I20250625 07:52:20.681947 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_786d6b63b6904ffe87ce1f3eb9769954" } modify_external_catalogs: true
I20250625 07:52:20.676072 713 leader_election.cc:304] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.682488 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.674132 1588 tablet.cc:1722] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:20.682981 1543 raft_consensus.cc:2802] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 07:52:20.683072 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.683146 1543 raft_consensus.cc:695] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:20.683401 1543 consensus_queue.cc:237] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:20.684053 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c50c35cac8e94e088cdacb664b8adfe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50c35cac8e94e088cdacb664b8adfe1 already in progress: creating tablet
I20250625 07:52:20.687814 1163 ts_tablet_manager.cc:1428] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.096s user 0.003s sys 0.000s
I20250625 07:52:20.694415 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e972e4cb0fb94531bc712f6133830d96" } new_table_name: "default.Table_579b087c06ea4011b8e8445e7ea3cd0e" modify_external_catalogs: true
I20250625 07:52:20.675541 757 tablet_service.cc:1468] Processing CreateTablet for tablet a7c6d068d2324ea6a07bb9be74578870 (DEFAULT_TABLE table=default.Table_82cd3da25b72491089bf529b94aafabd [id=3ff721dec5f145209210d0ae74e38b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.698696 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.699115 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:20.711414 1166 ts_tablet_manager.cc:1918] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.711894 1166 log.cc:1199] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/d92962b9e7d14c8d88d775b98e75de35
I20250625 07:52:20.712626 1166 ts_tablet_manager.cc:1939] T d92962b9e7d14c8d88d775b98e75de35 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:20.716169 1163 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.718015 1163 tablet_bootstrap.cc:654] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:20.725778 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f4bb74391a1c4e2792226d259a07f977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4bb74391a1c4e2792226d259a07f977 already in progress: creating tablet
W20250625 07:52:20.729331 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0b7daf759e324c1db853fdbaab76c721 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7daf759e324c1db853fdbaab76c721 already in progress: creating tablet
I20250625 07:52:20.729379 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet d92962b9e7d14c8d88d775b98e75de35 (table default.Table_2fb37c46511c4963a2b384669ec6331c [id=eecbc272d0844eaaaa51584b812f2321]) successfully deleted
W20250625 07:52:20.731725 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1973678168a84875b747897f08dd1bad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1973678168a84875b747897f08dd1bad already in progress: creating tablet
W20250625 07:52:20.733868 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 97b9a14151fc472e956126620633be32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b9a14151fc472e956126620633be32 already in progress: creating tablet
W20250625 07:52:20.735194 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7e60d4f9aed44082b9905fbb44ffb110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e60d4f9aed44082b9905fbb44ffb110 already in progress: creating tablet
W20250625 07:52:20.735991 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8bc8f47541c54c4287ebb7b892daa1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc8f47541c54c4287ebb7b892daa1be already in progress: creating tablet
W20250625 07:52:20.736084 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9578d77db00242d48f4e8f6f8263ef09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9578d77db00242d48f4e8f6f8263ef09 already in progress: creating tablet
W20250625 07:52:20.737066 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c50c35cac8e94e088cdacb664b8adfe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50c35cac8e94e088cdacb664b8adfe1 already in progress: creating tablet
I20250625 07:52:20.737263 1166 tablet_replica.cc:331] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:20.737517 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f4bb74391a1c4e2792226d259a07f977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4bb74391a1c4e2792226d259a07f977 already in progress: creating tablet
W20250625 07:52:20.738143 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8bc8f47541c54c4287ebb7b892daa1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc8f47541c54c4287ebb7b892daa1be already in progress: creating tablet
I20250625 07:52:20.738438 1163 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.738737 1163 ts_tablet_manager.cc:1397] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
W20250625 07:52:20.740262 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b1422987deca41a79f8db666d6961643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1422987deca41a79f8db666d6961643 already in progress: creating tablet
I20250625 07:52:20.740298 1163 raft_consensus.cc:357] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.740537 1163 raft_consensus.cc:383] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.740644 1163 raft_consensus.cc:738] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.740993 1163 consensus_queue.cc:260] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.746517 1158 tablet_bootstrap.cc:492] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:20.746778 1158 ts_tablet_manager.cc:1397] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.260s user 0.006s sys 0.002s
I20250625 07:52:20.747129 1160 ts_tablet_manager.cc:1918] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:20.747589 1160 log.cc:1199] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2a9945f38ef148caac59acaadcb2ef4c
I20250625 07:52:20.747951 1158 raft_consensus.cc:357] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.748236 1158 raft_consensus.cc:383] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.748363 1158 raft_consensus.cc:738] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.748733 1158 consensus_queue.cc:260] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.749145 1160 ts_tablet_manager.cc:1939] T 2a9945f38ef148caac59acaadcb2ef4c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:20.750783 1163 ts_tablet_manager.cc:1428] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250625 07:52:20.752974 1158 ts_tablet_manager.cc:1428] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:20.754238 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2a9945f38ef148caac59acaadcb2ef4c (table default.Table_4f645096238747a1a97bcd673b5438fd [id=836816eb909641daba87bdae20e0fd24]) successfully deleted
W20250625 07:52:20.754580 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3bed3f04c60249aaa389c26788974f58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bed3f04c60249aaa389c26788974f58 already in progress: creating tablet
W20250625 07:52:20.754884 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet faf049714a7e4d7eb947c29f272bbb7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faf049714a7e4d7eb947c29f272bbb7c already in progress: creating tablet
W20250625 07:52:20.755990 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5179a144a9a0445f933ca02e79bbc122 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5179a144a9a0445f933ca02e79bbc122 already in progress: creating tablet
I20250625 07:52:20.759088 1158 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:20.760510 1160 tablet_replica.cc:331] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:20.761399 1160 raft_consensus.cc:2241] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.762284 1160 raft_consensus.cc:2270] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.763473 1158 tablet_bootstrap.cc:654] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.764454 1160 ts_tablet_manager.cc:1905] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:20.774055 889 tablet_service.cc:1468] Processing CreateTablet for tablet 0cdee1535e5a4a21804e28e78aaea0eb (DEFAULT_TABLE table=default.Table_85849694e69e4737a9793e7c46e40a87 [id=fb8226e9c0a440629bae51d5425f6f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.774602 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdee1535e5a4a21804e28e78aaea0eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.775146 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96a1f823fd3b47c9b6d1cf5647f9a68a already in progress: creating tablet
I20250625 07:52:20.777047 893 tablet_service.cc:1468] Processing CreateTablet for tablet bfb421ea711a461890e267153250acea (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.777313 1163 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:20.777467 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb421ea711a461890e267153250acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.778426 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a (DEFAULT_TABLE table=default.Table_a976166b59b44d52a920b760df58965c [id=e43df540b1384567aec893aab717f2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.757877 754 tablet_service.cc:1468] Processing CreateTablet for tablet 0cdee1535e5a4a21804e28e78aaea0eb (DEFAULT_TABLE table=default.Table_85849694e69e4737a9793e7c46e40a87 [id=fb8226e9c0a440629bae51d5425f6f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.780076 749 tablet_service.cc:1468] Processing CreateTablet for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20 (DEFAULT_TABLE table=default.Table_404e6f3b9acf49a18305dbe0b2cac534 [id=bcaaae12d4874c77988476ec6df97320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.780531 743 tablet_service.cc:1468] Processing CreateTablet for tablet c00ad0924e234c9d889b4ef7c2048ddb (DEFAULT_TABLE table=default.Table_c43bae11f964428a83cb53dc9dee5bdf [id=ac101e3b4e40430ab6c2b76e707e5878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.780592 758 tablet_service.cc:1468] Processing CreateTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 (DEFAULT_TABLE table=default.Table_457927be3bfb45d1ab90fc5a3b8163d8 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.781100 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a (DEFAULT_TABLE table=default.Table_322b342d0f384574a0239ed582647def [id=981309d237144713a3eeeca6f8be1814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.781514 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b19f6c181ee4e24883b0d7df822ab3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.781445 756 tablet_service.cc:1468] Processing CreateTablet for tablet ef49f9f430824cf68fdd47e641039762 (DEFAULT_TABLE table=default.Table_dfa7845282e043bd8a70386031f91ff7 [id=ebe63b87ea394ecfa09158ca72688542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.783259 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 (DEFAULT_TABLE table=default.Table_9650a59cdcb04bf395e5cc36090a99f1 [id=b6b2da79cff84c8da09876da701889dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.783944 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.784794 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 (DEFAULT_TABLE table=default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.785005 895 tablet_service.cc:1468] Processing CreateTablet for tablet 73ec635d7d5242e8804dd3527457a540 (DEFAULT_TABLE table=default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379 [id=8e1c3b8709544ca2bdd8849e5fca84f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.785378 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec635d7d5242e8804dd3527457a540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.788151 742 tablet_service.cc:1468] Processing CreateTablet for tablet 550e64a19120444a8e7b27db9bbf09f5 (DEFAULT_TABLE table=default.Table_d3a5e89e18914595b2f605c893c2d050 [id=1a2f0c0bdaa54a92b0d8f5eae1462460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.788977 885 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a (DEFAULT_TABLE table=default.Table_a976166b59b44d52a920b760df58965c [id=e43df540b1384567aec893aab717f2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.789325 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5a40faa418420ea31bc1d46cab1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.789294 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a (DEFAULT_TABLE table=default.Table_322b342d0f384574a0239ed582647def [id=981309d237144713a3eeeca6f8be1814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.790278 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20 (DEFAULT_TABLE table=default.Table_404e6f3b9acf49a18305dbe0b2cac534 [id=bcaaae12d4874c77988476ec6df97320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.791643 1166 raft_consensus.cc:2241] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:20.792124 1166 raft_consensus.cc:2270] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:20.792878 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 73ec635d7d5242e8804dd3527457a540 (DEFAULT_TABLE table=default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379 [id=8e1c3b8709544ca2bdd8849e5fca84f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.792996 880 tablet_service.cc:1468] Processing CreateTablet for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20 (DEFAULT_TABLE table=default.Table_404e6f3b9acf49a18305dbe0b2cac534 [id=bcaaae12d4874c77988476ec6df97320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.793350 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.794324 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a (DEFAULT_TABLE table=default.Table_a976166b59b44d52a920b760df58965c [id=e43df540b1384567aec893aab717f2c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.794500 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c00ad0924e234c9d889b4ef7c2048ddb (DEFAULT_TABLE table=default.Table_c43bae11f964428a83cb53dc9dee5bdf [id=ac101e3b4e40430ab6c2b76e707e5878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.794644 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 (DEFAULT_TABLE table=default.Table_457927be3bfb45d1ab90fc5a3b8163d8 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.795791 1027 tablet_service.cc:1468] Processing CreateTablet for tablet ef49f9f430824cf68fdd47e641039762 (DEFAULT_TABLE table=default.Table_dfa7845282e043bd8a70386031f91ff7 [id=ebe63b87ea394ecfa09158ca72688542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.796865 1163 tablet_bootstrap.cc:654] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.797005 890 tablet_service.cc:1468] Processing CreateTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 (DEFAULT_TABLE table=default.Table_457927be3bfb45d1ab90fc5a3b8163d8 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.797355 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b5bb3fe3ef4bfdba93b862784a9f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.800169 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 (DEFAULT_TABLE table=default.Table_a2517172003a44c492a3e3b9260bc0d5 [id=56cf985cdf85438f847e59f7f116e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.801143 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b19f6c181ee4e24883b0d7df822ab3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.801674 883 tablet_service.cc:1468] Processing CreateTablet for tablet c00ad0924e234c9d889b4ef7c2048ddb (DEFAULT_TABLE table=default.Table_c43bae11f964428a83cb53dc9dee5bdf [id=ac101e3b4e40430ab6c2b76e707e5878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.801999 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00ad0924e234c9d889b4ef7c2048ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.803094 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a80d81cc2d524f318b858a9bd97a89b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.805001 884 tablet_service.cc:1468] Processing CreateTablet for tablet ef49f9f430824cf68fdd47e641039762 (DEFAULT_TABLE table=default.Table_dfa7845282e043bd8a70386031f91ff7 [id=ebe63b87ea394ecfa09158ca72688542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.805328 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef49f9f430824cf68fdd47e641039762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.809037 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00ad0924e234c9d889b4ef7c2048ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.809065 886 tablet_service.cc:1468] Processing CreateTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 (DEFAULT_TABLE table=default.Table_a2517172003a44c492a3e3b9260bc0d5 [id=56cf985cdf85438f847e59f7f116e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.809393 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a80d81cc2d524f318b858a9bd97a89b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.758690 744 tablet_service.cc:1468] Processing CreateTablet for tablet bfb421ea711a461890e267153250acea (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.759512 746 tablet_service.cc:1468] Processing CreateTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a (DEFAULT_TABLE table=default.Table_322b342d0f384574a0239ed582647def [id=981309d237144713a3eeeca6f8be1814]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.814368 1581 mvcc.cc:204] Tried to move back new op lower bound from 7171432200301363200 to 7171432195927900160. Current Snapshot: MvccSnapshot[applied={T|T < 7171432200301363200}]
I20250625 07:52:20.814639 1588 tablet.cc:1717] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:20.819571 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5a40faa418420ea31bc1d46cab1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.819144 1012 tablet_service.cc:1468] Processing CreateTablet for tablet bfb421ea711a461890e267153250acea (DEFAULT_TABLE table=default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.842175 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef49f9f430824cf68fdd47e641039762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.847568 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.856781 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b5bb3fe3ef4bfdba93b862784a9f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.859000 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec635d7d5242e8804dd3527457a540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.874441 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 (DEFAULT_TABLE table=default.Table_9650a59cdcb04bf395e5cc36090a99f1 [id=b6b2da79cff84c8da09876da701889dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.879156 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.879622 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.879848 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb421ea711a461890e267153250acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.765044 751 tablet_service.cc:1468] Processing CreateTablet for tablet 962633b0d9644c308f0b7fe1db714daf (DEFAULT_TABLE table=default.Table_5fbde73edcc941ebb33753389aea2dc8 [id=a01d744167054a7bb92fd9a2eee73d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.765810 1543 raft_consensus.cc:491] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.888674 1543 raft_consensus.cc:513] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.889461 1543 leader_election.cc:290] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.778447 745 tablet_service.cc:1468] Processing CreateTablet for tablet 73ec635d7d5242e8804dd3527457a540 (DEFAULT_TABLE table=default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379 [id=8e1c3b8709544ca2bdd8849e5fca84f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.801046 747 tablet_service.cc:1468] Processing CreateTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 (DEFAULT_TABLE table=default.Table_a2517172003a44c492a3e3b9260bc0d5 [id=56cf985cdf85438f847e59f7f116e3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.754637 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6d068d2324ea6a07bb9be74578870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.804875 1162 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:20.891255 1162 ts_tablet_manager.cc:1397] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.277s user 0.009s sys 0.002s
I20250625 07:52:20.891341 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ec635d7d5242e8804dd3527457a540. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.891466 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76fe692067954856908f936ced332c18" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:20.891624 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef49f9f430824cf68fdd47e641039762. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:20.892064 713 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.892225 1162 raft_consensus.cc:357] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.892445 1162 raft_consensus.cc:383] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:20.892558 1162 raft_consensus.cc:738] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:20.892841 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b5bb3fe3ef4bfdba93b862784a9f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.893637 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5a40faa418420ea31bc1d46cab1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.896874 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a80d81cc2d524f318b858a9bd97a89b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.897125 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cdee1535e5a4a21804e28e78aaea0eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.897240 1162 consensus_queue.cc:260] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.897852 1162 ts_tablet_manager.cc:1428] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:20.898365 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75 (DEFAULT_TABLE table=default.Table_e6509d5448ab4dba96e3a6571d4963a7 [id=b0330ed5b54243c983112b2afaf66f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.898629 1162 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:20.898948 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.900532 1162 tablet_bootstrap.cc:654] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:20.904170 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fe5a9156b14d1ea79fd1f3a04dcd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.908663 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f655669b74b4a6594fbd63c45d15bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.911677 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76fe692067954856908f936ced332c18" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:20.912623 713 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Illegal state: Tablet not RUNNING: STOPPED
I20250625 07:52:20.912818 713 leader_election.cc:304] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.913332 1543 raft_consensus.cc:2747] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.914754 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]). Expected version 2 got 0
I20250625 07:52:20.914845 1582 raft_consensus.cc:491] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.914996 1582 raft_consensus.cc:513] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.916532 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0024618293d74f559b04258f66616044" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:20.918126 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 (DEFAULT_TABLE table=default.Table_9650a59cdcb04bf395e5cc36090a99f1 [id=b6b2da79cff84c8da09876da701889dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.918994 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_40088cb2328f4d629da1e5d5341d135d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:20.919245 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f655669b74b4a6594fbd63c45d15bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.919629 588 catalog_manager.cc:5582] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
W20250625 07:52:20.924192 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40088cb2328f4d629da1e5d5341d135d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:20.929483 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b19f6c181ee4e24883b0d7df822ab3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.930706 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0024618293d74f559b04258f66616044" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.934517 850 leader_election.cc:343] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.940323 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb421ea711a461890e267153250acea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.943404 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e64a19120444a8e7b27db9bbf09f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.944156 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f655669b74b4a6594fbd63c45d15bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.944876 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962633b0d9644c308f0b7fe1db714daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.946429 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e77ff1b8ca848ff8e1b4538645b4be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.946949 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a29a9ba27ab4eb494e1b91e4b33aa75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.953136 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 (DEFAULT_TABLE table=default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.953824 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e77ff1b8ca848ff8e1b4538645b4be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.955427 889 tablet_service.cc:1468] Processing CreateTablet for tablet 550e64a19120444a8e7b27db9bbf09f5 (DEFAULT_TABLE table=default.Table_d3a5e89e18914595b2f605c893c2d050 [id=1a2f0c0bdaa54a92b0d8f5eae1462460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.955984 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e64a19120444a8e7b27db9bbf09f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.956840 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:20.961223 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c00ad0924e234c9d889b4ef7c2048ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.968564 1582 leader_election.cc:290] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:20.968876 851 leader_election.cc:343] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.969254 851 leader_election.cc:304] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.969682 1582 raft_consensus.cc:2747] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.974814 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 (DEFAULT_TABLE table=default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.975263 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e77ff1b8ca848ff8e1b4538645b4be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.977854 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 550e64a19120444a8e7b27db9bbf09f5 (DEFAULT_TABLE table=default.Table_d3a5e89e18914595b2f605c893c2d050 [id=1a2f0c0bdaa54a92b0d8f5eae1462460]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:20.978256 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e64a19120444a8e7b27db9bbf09f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:20.979539 1582 raft_consensus.cc:491] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.979691 1582 raft_consensus.cc:513] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.980432 1582 leader_election.cc:290] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.981176 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a777f64425384be5bccc86b197626618" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:20.981827 851 leader_election.cc:343] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.985302 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a777f64425384be5bccc86b197626618" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.985914 850 leader_election.cc:343] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.986066 850 leader_election.cc:304] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.986759 1543 raft_consensus.cc:491] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:20.986925 1543 raft_consensus.cc:513] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.987558 1543 leader_election.cc:290] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:20.988317 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c770b41e0844786ad81c7b3579f9934" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:20.989651 713 leader_election.cc:343] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:20.995009 1582 raft_consensus.cc:2747] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:20.995647 713 leader_election.cc:343] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:20.995801 713 leader_election.cc:304] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:20.996328 1543 raft_consensus.cc:2747] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:20.997085 1163 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:20.997248 1163 ts_tablet_manager.cc:1397] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.220s user 0.026s sys 0.044s
I20250625 07:52:20.998551 1163 raft_consensus.cc:357] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:20.998775 1163 raft_consensus.cc:383] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.004048 1158 tablet_bootstrap.cc:492] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.004211 1158 ts_tablet_manager.cc:1397] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.250s user 0.021s sys 0.038s
I20250625 07:52:21.004994 1543 raft_consensus.cc:491] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.005158 1543 raft_consensus.cc:513] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.005882 1543 leader_election.cc:290] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.006721 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ebb943df93284913b6a520e099399322" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:21.007558 713 leader_election.cc:343] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.007516 1158 raft_consensus.cc:357] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.007680 1158 raft_consensus.cc:383] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.007772 1158 raft_consensus.cc:738] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.008008 1158 consensus_queue.cc:260] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.008555 1158 ts_tablet_manager.cc:1428] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:21.008747 1163 raft_consensus.cc:738] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:21.009140 713 leader_election.cc:343] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ebb943df93284913b6a520e099399322
I20250625 07:52:21.009049 1163 consensus_queue.cc:260] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.009352 1158 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.009295 713 leader_election.cc:304] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.009999 1163 ts_tablet_manager.cc:1428] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250625 07:52:21.010849 1160 ts_tablet_manager.cc:1918] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.011044 1158 tablet_bootstrap.cc:654] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.015771 1543 raft_consensus.cc:2747] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:21.017731 1166 ts_tablet_manager.cc:1905] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.020613 1163 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.023137 1160 log.cc:1199] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/76fe692067954856908f936ced332c18
I20250625 07:52:21.023561 1160 ts_tablet_manager.cc:1939] T 76fe692067954856908f936ced332c18 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.024338 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 500) took 1009 ms. Trace:
I20250625 07:52:21.024446 1160 rpcz_store.cc:276] 0625 07:52:20.014518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.016885 (+ 2367us) service_pool.cc:225] Handling call
0625 07:52:20.760250 (+743365us) ts_tablet_manager.cc:1145] Deleting tablet 76fe692067954856908f936ced332c18
0625 07:52:20.760264 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:20.991138 (+230874us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.010735 (+ 19597us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.024267 (+ 13532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":743058}
I20250625 07:52:21.024989 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 501) took 1007 ms. Trace:
I20250625 07:52:21.025075 1160 rpcz_store.cc:276] 0625 07:52:20.017681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.017739 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:21.024810 (+1007071us) ts_tablet_manager.cc:1145] Deleting tablet 1898ff5b7626478bb997f7a433477498
0625 07:52:21.024819 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.024932 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006876}
I20250625 07:52:21.025367 1160 tablet_replica.cc:331] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.025560 1160 raft_consensus.cc:2241] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.026789 1158 tablet_bootstrap.cc:492] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.026965 1158 ts_tablet_manager.cc:1397] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20250625 07:52:21.027735 1158 raft_consensus.cc:357] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.027940 1158 raft_consensus.cc:383] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.028015 1158 raft_consensus.cc:738] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.028236 1158 consensus_queue.cc:260] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.029145 1158 ts_tablet_manager.cc:1428] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:21.029579 1158 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.030805 593 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c2fef3a67351440b816ba1cb73b85774
I20250625 07:52:21.032328 1158 tablet_bootstrap.cc:654] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.033488 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e8fef495e82e483589a4868b7093361c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.034117 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8fef495e82e483589a4868b7093361c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.037276 1163 tablet_bootstrap.cc:654] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.040637 1160 raft_consensus.cc:2270] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.042178 1160 ts_tablet_manager.cc:1905] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.045423 1163 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.045665 1163 ts_tablet_manager.cc:1397] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20250625 07:52:21.046501 1163 raft_consensus.cc:357] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.046772 1163 raft_consensus.cc:383] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.046895 1163 raft_consensus.cc:738] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.047216 1163 consensus_queue.cc:260] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.052350 1163 ts_tablet_manager.cc:1428] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:21.058393 1163 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.060420 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5b19f6c181ee4e24883b0d7df822ab3a
I20250625 07:52:21.060463 1163 tablet_bootstrap.cc:654] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.062022 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 5de1492e483e47919ee5581ecee227be (table default.Table_0ef24d5b4b124360b654d6903b994151 [id=23322aa9321140c39272b5b2a102fb02]). Expected version 2 got 0
I20250625 07:52:21.063115 590 master_service.cc:948] ReplaceTablet: received request to replace tablet fe630469fe7c4f4bac28277896eddc9c from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.066987 1160 ts_tablet_manager.cc:1918] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.067824 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.068645 1160 log.cc:1199] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:21.069265 1160 ts_tablet_manager.cc:1939] T c7a2b94af59648239cfd1e5ce5e8b968 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.070748 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 502) took 1052 ms. Trace:
I20250625 07:52:21.070981 1160 rpcz_store.cc:276] 0625 07:52:20.017812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.017971 (+ 159us) service_pool.cc:225] Handling call
0625 07:52:21.025291 (+1007320us) ts_tablet_manager.cc:1145] Deleting tablet c7a2b94af59648239cfd1e5ce5e8b968
0625 07:52:21.025298 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.063409 (+ 38111us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.066854 (+ 3445us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.070689 (+ 3835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007142}
I20250625 07:52:21.071609 1160 tablet_replica.cc:331] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.071866 1160 raft_consensus.cc:2241] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.072085 1160 raft_consensus.cc:2270] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.072661 1163 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.072857 1163 ts_tablet_manager.cc:1397] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20250625 07:52:21.073127 1160 ts_tablet_manager.cc:1905] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.073743 1163 raft_consensus.cc:357] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.074471 1163 raft_consensus.cc:383] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.074620 1163 raft_consensus.cc:738] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.074985 1163 consensus_queue.cc:260] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.075505 1163 ts_tablet_manager.cc:1428] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:21.075928 1163 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.077764 1163 tablet_bootstrap.cc:654] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.082953 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 76fe692067954856908f936ced332c18 (table default.Table_33aedac05c3649c0a21c0367c4ff59a3 [id=b02af454e5f6454885578ace6c10ad97]) successfully deleted
W20250625 07:52:21.083256 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1898ff5b7626478bb997f7a433477498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1898ff5b7626478bb997f7a433477498 already in progress: creating tablet
I20250625 07:52:21.083477 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c7a2b94af59648239cfd1e5ce5e8b968 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:21.084149 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b71aadf074a4a39943b2ff930909f93 with delete_type TABLET_DATA_DELETED (replaced by tablet 9cc1842f257d409ba0a56873f582f023) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.084920 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e800d4982fa45f4b931985c8b52719d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.085450 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet c81396ee6ed7400baa30dcc5d71ba45d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.085958 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.086463 1161 tablet_replica.cc:331] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.086649 1161 raft_consensus.cc:2241] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.086828 1161 raft_consensus.cc:2270] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.087334 1161 ts_tablet_manager.cc:1905] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.092981 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:21.093813 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9936302439324d268bee80fe82d75dd4" } new_table_name: "default.Table_2b519d271b96497dafe0b274a8755066" modify_external_catalogs: true
I20250625 07:52:21.099745 746 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.099860 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:21.104876 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.109323 1161 ts_tablet_manager.cc:1918] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.109555 1161 log.cc:1199] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6b71aadf074a4a39943b2ff930909f93
I20250625 07:52:21.110689 1166 ts_tablet_manager.cc:1918] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.110852 1166 log.cc:1199] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1e6c2988baa14cbc8c878f79e942880d
I20250625 07:52:21.111302 1166 ts_tablet_manager.cc:1939] T 1e6c2988baa14cbc8c878f79e942880d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.111455 1161 ts_tablet_manager.cc:1939] T 6b71aadf074a4a39943b2ff930909f93 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:21.112545 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1e6c2988baa14cbc8c878f79e942880d (table default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]) successfully deleted
I20250625 07:52:21.113945 1162 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:21.113853 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3666ef79a51b4d33a802675181e67fac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.114509 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3666ef79a51b4d33a802675181e67fac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.115796 1162 ts_tablet_manager.cc:1397] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.218s user 0.045s sys 0.029s
I20250625 07:52:21.117036 1161 tablet_replica.cc:331] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.117067 1162 raft_consensus.cc:357] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.117348 1161 raft_consensus.cc:2241] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.117574 1161 raft_consensus.cc:2270] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.118685 1161 ts_tablet_manager.cc:1905] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.119171 1162 raft_consensus.cc:383] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.119565 1162 raft_consensus.cc:738] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.120097 1162 consensus_queue.cc:260] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.121448 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9ede02c06bf540edbff83b6d2e68d206"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.121999 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ede02c06bf540edbff83b6d2e68d206 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.121603 1162 ts_tablet_manager.cc:1428] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:21.123344 1162 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:21.123987 1160 ts_tablet_manager.cc:1918] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.124925 1160 log.cc:1199] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:21.125363 1160 ts_tablet_manager.cc:1939] T e7100ef72ff14c24be92c21c6eb8a9de P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.125944 1162 tablet_bootstrap.cc:654] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.126870 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 503) took 1108 ms. Trace:
W20250625 07:52:21.128036 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 019959e9abed46fbbbb4755cb01cef44 consensus state has no leader
I20250625 07:52:21.131984 1160 rpcz_store.cc:276] 0625 07:52:20.017974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.018694 (+ 720us) service_pool.cc:225] Handling call
0625 07:52:21.071538 (+1052844us) ts_tablet_manager.cc:1145] Deleting tablet e7100ef72ff14c24be92c21c6eb8a9de
0625 07:52:21.071548 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.114037 (+ 42489us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.123857 (+ 9820us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.126797 (+ 2940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052661}
I20250625 07:52:21.132740 1160 tablet_replica.cc:331] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.133625 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e8d65c2ed5240fc923a7c7b59488e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.133442 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.133875 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6b71aadf074a4a39943b2ff930909f93 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:21.134440 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.134472 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e7100ef72ff14c24be92c21c6eb8a9de (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
I20250625 07:52:21.135028 884 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.135083 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.135452 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.134135 1160 raft_consensus.cc:2241] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.136220 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a0e01d2875e4b639d653e035aa60679" } new_table_name: "default.Table_c4ec962d89484a528ecd0f46a281c91b" modify_external_catalogs: true
I20250625 07:52:21.139030 1160 raft_consensus.cc:2270] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.139678 1160 ts_tablet_manager.cc:1905] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.142264 1161 ts_tablet_manager.cc:1918] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.142565 1161 log.cc:1199] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9e800d4982fa45f4b931985c8b52719d
I20250625 07:52:21.143118 1161 ts_tablet_manager.cc:1939] T 9e800d4982fa45f4b931985c8b52719d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:21.143522 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:21.145048 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.148036 1161 tablet_replica.cc:331] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.151338 1160 ts_tablet_manager.cc:1918] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.151468 1160 log.cc:1199] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5c28a064006145eea000e74da10d0794
I20250625 07:52:21.151680 1161 raft_consensus.cc:2241] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.151902 1161 raft_consensus.cc:2270] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.152372 1161 ts_tablet_manager.cc:1905] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.153146 1160 ts_tablet_manager.cc:1939] T 5c28a064006145eea000e74da10d0794 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.154016 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 505) took 1135 ms. Trace:
I20250625 07:52:21.154193 1160 rpcz_store.cc:276] 0625 07:52:20.018238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.018901 (+ 663us) service_pool.cc:225] Handling call
0625 07:52:21.132619 (+1113718us) ts_tablet_manager.cc:1145] Deleting tablet 5c28a064006145eea000e74da10d0794
0625 07:52:21.132631 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.148837 (+ 16206us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.151222 (+ 2385us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.153967 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113567}
I20250625 07:52:21.154829 1160 tablet_replica.cc:331] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.155082 1160 raft_consensus.cc:2241] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.155328 1160 raft_consensus.cc:2270] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.155964 1160 ts_tablet_manager.cc:1905] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.159340 1161 ts_tablet_manager.cc:1918] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.159550 1161 log.cc:1199] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c81396ee6ed7400baa30dcc5d71ba45d
I20250625 07:52:21.159731 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3921e8fba62d419b997de9a801f2274a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.160164 1161 ts_tablet_manager.cc:1939] T c81396ee6ed7400baa30dcc5d71ba45d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:21.160529 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3921e8fba62d419b997de9a801f2274a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.164311 1160 ts_tablet_manager.cc:1918] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.164510 1160 log.cc:1199] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:21.164995 1160 ts_tablet_manager.cc:1939] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.165751 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 504) took 1147 ms. Trace:
I20250625 07:52:21.165876 1160 rpcz_store.cc:276] 0625 07:52:20.018098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.022706 (+ 4608us) service_pool.cc:225] Handling call
0625 07:52:21.154765 (+1132059us) ts_tablet_manager.cc:1145] Deleting tablet 29eaa8d93e5b4d1d8ed147b60c7da859
0625 07:52:21.154776 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.159562 (+ 4786us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.164193 (+ 4631us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:21.165695 (+ 1502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131839}
I20250625 07:52:21.166340 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 509) took 1070 ms. Trace:
I20250625 07:52:21.166448 1160 rpcz_store.cc:276] 0625 07:52:20.095981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:20.096034 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:21.166175 (+1070141us) ts_tablet_manager.cc:1145] Deleting tablet f191436f3b9244eaa12d211b6f125175
0625 07:52:21.166184 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:21.166278 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069884}
I20250625 07:52:21.167779 1160 tablet_replica.cc:331] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.167995 1160 raft_consensus.cc:2241] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.168195 1160 raft_consensus.cc:2270] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.168763 1160 ts_tablet_manager.cc:1905] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:21.169030 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e has no consensus state
I20250625 07:52:21.170537 1163 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.171181 1158 tablet_bootstrap.cc:492] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.171500 1158 ts_tablet_manager.cc:1397] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.142s user 0.024s sys 0.014s
I20250625 07:52:21.172609 1158 raft_consensus.cc:357] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.172878 1163 ts_tablet_manager.cc:1397] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.014s
I20250625 07:52:21.173759 1163 raft_consensus.cc:357] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.174005 1158 raft_consensus.cc:383] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.174161 1163 raft_consensus.cc:383] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.174294 1158 raft_consensus.cc:738] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.174592 1158 consensus_queue.cc:260] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.174824 1163 raft_consensus.cc:738] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.175630 1158 ts_tablet_manager.cc:1428] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:21.175104 1163 consensus_queue.cc:260] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.176156 1163 ts_tablet_manager.cc:1428] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:21.176487 1158 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.178293 1158 tablet_bootstrap.cc:654] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.178530 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5c28a064006145eea000e74da10d0794 (table default.Table_9bbce7454a884554b31faffc79a33b99 [id=cbb5aafb8a9243d98c7d8578865c13b9]) successfully deleted
I20250625 07:52:21.178812 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (table default.Table_97384ebbc1584955af0a6bc425084522 [id=131a7d8cbeb34a0a8db53be305d31ccd]) successfully deleted
W20250625 07:52:21.179042 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f191436f3b9244eaa12d211b6f125175 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f191436f3b9244eaa12d211b6f125175 already in progress: creating tablet
I20250625 07:52:21.179122 1163 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:21.179257 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7e60d4f9aed44082b9905fbb44ffb110 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e60d4f9aed44082b9905fbb44ffb110 already in progress: creating tablet
W20250625 07:52:21.179445 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9578d77db00242d48f4e8f6f8263ef09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9578d77db00242d48f4e8f6f8263ef09 already in progress: creating tablet
I20250625 07:52:21.179579 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:21.179631 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c50c35cac8e94e088cdacb664b8adfe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c50c35cac8e94e088cdacb664b8adfe1 already in progress: creating tablet
W20250625 07:52:21.179848 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1973678168a84875b747897f08dd1bad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1973678168a84875b747897f08dd1bad already in progress: creating tablet
W20250625 07:52:21.180056 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f4bb74391a1c4e2792226d259a07f977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4bb74391a1c4e2792226d259a07f977 already in progress: creating tablet
W20250625 07:52:21.180462 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0b7daf759e324c1db853fdbaab76c721 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7daf759e324c1db853fdbaab76c721 already in progress: creating tablet
I20250625 07:52:21.181288 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d50dc89e611a4cd490d628b4a824a34f" } new_table_name: "default.Table_d77e4cd0cb1143258e09219d7910fc82" modify_external_catalogs: true
I20250625 07:52:21.181437 1163 tablet_bootstrap.cc:654] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.176281 1160 ts_tablet_manager.cc:1918] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.182695 1160 log.cc:1199] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a777f64425384be5bccc86b197626618
I20250625 07:52:21.183094 1160 ts_tablet_manager.cc:1939] T a777f64425384be5bccc86b197626618 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.185213 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.185498 884 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.187314 1333 consensus_queue.cc:1035] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:21.187541 1160 tablet_replica.cc:331] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.187767 1160 raft_consensus.cc:2241] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.187943 1160 raft_consensus.cc:2270] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.188596 1160 ts_tablet_manager.cc:1905] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.189215 1161 tablet_replica.cc:331] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.189412 1161 raft_consensus.cc:2241] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.189637 1161 raft_consensus.cc:2270] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.191519 1161 ts_tablet_manager.cc:1905] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:21.191956 713 consensus_peers.cc:487] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:21.193323 1166 tablet_replica.cc:331] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.193770 1166 raft_consensus.cc:2241] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.193960 1166 raft_consensus.cc:2270] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.194588 1166 ts_tablet_manager.cc:1905] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:21.199344 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0b7daf759e324c1db853fdbaab76c721 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7daf759e324c1db853fdbaab76c721 already in progress: creating tablet
W20250625 07:52:21.199761 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b1422987deca41a79f8db666d6961643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1422987deca41a79f8db666d6961643 already in progress: creating tablet
I20250625 07:52:21.203078 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6a808c6d46a94091b18c21073a499b8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.203681 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a808c6d46a94091b18c21073a499b8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.205816 1161 ts_tablet_manager.cc:1918] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:21.180660 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8bc8f47541c54c4287ebb7b892daa1be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bc8f47541c54c4287ebb7b892daa1be already in progress: creating tablet
W20250625 07:52:21.210975 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b1422987deca41a79f8db666d6961643 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1422987deca41a79f8db666d6961643 already in progress: creating tablet
W20250625 07:52:21.211189 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 97b9a14151fc472e956126620633be32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b9a14151fc472e956126620633be32 already in progress: creating tablet
I20250625 07:52:21.211373 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a777f64425384be5bccc86b197626618 (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
I20250625 07:52:21.211570 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9e800d4982fa45f4b931985c8b52719d (table default.Table_98b635139b69481f882d18773584a8e7 [id=3c6df96cbd16451faac3629abef20843]) successfully deleted
I20250625 07:52:21.211772 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c81396ee6ed7400baa30dcc5d71ba45d (table default.Table_3aec9b6017944d5f8293424556b380d3 [id=975c0e95149a474b8e98ad4c4411b385]) successfully deleted
I20250625 07:52:21.211360 1485 raft_consensus.cc:491] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.215267 1485 raft_consensus.cc:513] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.216187 1485 leader_election.cc:290] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:21.215267 1161 log.cc:1199] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8e8d65c2ed5240fc923a7c7b59488e6a
I20250625 07:52:21.217598 1161 ts_tablet_manager.cc:1939] T 8e8d65c2ed5240fc923a7c7b59488e6a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:21.220268 985 leader_election.cc:343] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b769ac57346a427abd22214eebc0d324
W20250625 07:52:21.211971 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5b19f6c181ee4e24883b0d7df822ab3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b19f6c181ee4e24883b0d7df822ab3a already in progress: creating tablet
I20250625 07:52:21.230907 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet b769ac57346a427abd22214eebc0d324 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.231071 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8e8d65c2ed5240fc923a7c7b59488e6a (table default.Table_5454db9f8f894091981c0f6abdfd2955 [id=5c7e0fd8cbcd4cbbb6b5926abc27ce4b]) successfully deleted
I20250625 07:52:21.231335 1161 tablet_replica.cc:331] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:21.231353 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet cf175e17865e40ceb26aa955b91c2c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf175e17865e40ceb26aa955b91c2c13 already in progress: creating tablet
W20250625 07:52:21.231587 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 10c7f4b4187c428bbec136c85219da4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10c7f4b4187c428bbec136c85219da4d already in progress: creating tablet
I20250625 07:52:21.231616 1161 raft_consensus.cc:2241] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:21.231812 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 960ded22e2eb42d9a3494c020a4ea354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960ded22e2eb42d9a3494c020a4ea354 already in progress: creating tablet
I20250625 07:52:21.232043 1161 raft_consensus.cc:2270] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.233000 1161 ts_tablet_manager.cc:1905] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.233147 1160 ts_tablet_manager.cc:1918] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.235617 1160 log.cc:1199] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0024618293d74f559b04258f66616044
I20250625 07:52:21.238085 1160 ts_tablet_manager.cc:1939] T 0024618293d74f559b04258f66616044 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.241377 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e342d843577548959d8be6a80119e769"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.242048 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e342d843577548959d8be6a80119e769 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.245575 1163 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.245762 1163 ts_tablet_manager.cc:1397] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.009s
I20250625 07:52:21.246331 1158 tablet_bootstrap.cc:492] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.246376 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0024618293d74f559b04258f66616044 (table default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]) successfully deleted
I20250625 07:52:21.247054 1333 raft_consensus.cc:491] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.247071 1163 raft_consensus.cc:357] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.247242 1163 raft_consensus.cc:383] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.247221 1333 raft_consensus.cc:513] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.247313 1163 raft_consensus.cc:738] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.247514 1163 consensus_queue.cc:260] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.247965 1163 ts_tablet_manager.cc:1428] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:21.248224 1163 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.249357 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.250136 1610 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:21.252516 1160 tablet_replica.cc:331] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.252837 1160 raft_consensus.cc:2241] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.253839 1160 raft_consensus.cc:2270] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:21.253906 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5b19f6c181ee4e24883b0d7df822ab3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b19f6c181ee4e24883b0d7df822ab3a already in progress: creating tablet
I20250625 07:52:21.254508 1160 ts_tablet_manager.cc:1905] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:21.255764 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet cf175e17865e40ceb26aa955b91c2c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf175e17865e40ceb26aa955b91c2c13 already in progress: creating tablet
W20250625 07:52:21.255999 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 10c7f4b4187c428bbec136c85219da4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10c7f4b4187c428bbec136c85219da4d already in progress: creating tablet
W20250625 07:52:21.256251 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 960ded22e2eb42d9a3494c020a4ea354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960ded22e2eb42d9a3494c020a4ea354 already in progress: creating tablet
I20250625 07:52:21.251300 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c0edbf6cbdb49a6b3d3fff7f347c8ce" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:21.251797 986 leader_election.cc:343] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b769ac57346a427abd22214eebc0d324
I20250625 07:52:21.258184 986 leader_election.cc:304] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:21.258412 986 raft_consensus.cc:2684] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:21.259101 1158 ts_tablet_manager.cc:1397] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.010s
I20250625 07:52:21.262099 1158 raft_consensus.cc:357] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.264633 1158 raft_consensus.cc:383] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.265173 1158 raft_consensus.cc:738] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.265867 1158 consensus_queue.cc:260] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.270860 1333 leader_election.cc:290] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.272164 1609 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:21.274811 1158 ts_tablet_manager.cc:1428] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250625 07:52:21.275168 1333 consensus_queue.cc:1035] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250625 07:52:21.279242 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fe630469fe7c4f4bac28277896eddc9c
I20250625 07:52:21.279779 590 catalog_manager.cc:6652] ReplaceTablet: tablet fe630469fe7c4f4bac28277896eddc9c deleted and replaced by tablet f815ed67e17347dcad804cbc1ba7660c
W20250625 07:52:21.279940 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 60864.5us
W20250625 07:52:21.280884 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8c015c202544914b74f1e245b496f82 consensus state has no leader
I20250625 07:52:21.281150 594 master_service.cc:948] ReplaceTablet: received request to replace tablet e06ecad6c7464ffe8ec6f632634307c9 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.282400 1158 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.284173 1158 tablet_bootstrap.cc:654] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.285813 1163 tablet_bootstrap.cc:654] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.287169 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.289860 884 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.290285 1048 raft_consensus.cc:3058] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:21.293844 1161 ts_tablet_manager.cc:1918] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.299579 1161 log.cc:1199] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b769ac57346a427abd22214eebc0d324
I20250625 07:52:21.302412 1163 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.302613 1163 ts_tablet_manager.cc:1397] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20250625 07:52:21.303436 1163 raft_consensus.cc:357] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.303676 1163 raft_consensus.cc:383] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.303754 1163 raft_consensus.cc:738] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.304024 1163 consensus_queue.cc:260] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.304518 1163 ts_tablet_manager.cc:1428] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:21.307258 1588 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:21.308161 1163 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.311249 1160 ts_tablet_manager.cc:1918] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.311399 1160 log.cc:1199] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:21.311798 1160 ts_tablet_manager.cc:1939] T c65f257ae653488a8b49b0b523d2cf82 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.312247 1163 tablet_bootstrap.cc:654] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.313534 1161 ts_tablet_manager.cc:1939] T b769ac57346a427abd22214eebc0d324 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:21.314671 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b769ac57346a427abd22214eebc0d324 (table default.Table_08a3260b659949be80df2e4a6bd81cad [id=1dd98ab8da6243468e3439bdeea865eb]) successfully deleted
I20250625 07:52:21.317364 755 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:21.317677 713 leader_election.cc:343] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:21.318015 713 consensus_peers.cc:487] T 4914a64e704a4386b8241e95229c3786 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:21.318766 713 leader_election.cc:343] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3c0edbf6cbdb49a6b3d3fff7f347c8ce
I20250625 07:52:21.318933 713 leader_election.cc:304] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.319330 1616 raft_consensus.cc:2747] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:21.319962 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fe630469fe7c4f4bac28277896eddc9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe630469fe7c4f4bac28277896eddc9c already in progress: creating tablet
I20250625 07:52:21.320946 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c65f257ae653488a8b49b0b523d2cf82 (table default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]) successfully deleted
W20250625 07:52:21.321550 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fe630469fe7c4f4bac28277896eddc9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe630469fe7c4f4bac28277896eddc9c already in progress: creating tablet
I20250625 07:52:21.327059 1162 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:21.327432 1162 ts_tablet_manager.cc:1397] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.022s
I20250625 07:52:21.328323 1162 raft_consensus.cc:357] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.327235 1617 raft_consensus.cc:491] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.331884 1162 raft_consensus.cc:383] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.334064 1617 raft_consensus.cc:513] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.334923 1162 raft_consensus.cc:738] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.336725 1617 leader_election.cc:290] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.337064 1162 consensus_queue.cc:260] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.337885 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f62b03db94d4ee3935866b5dd8bd53f" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:21.338150 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu pid 1268
W20250625 07:52:21.338656 713 leader_election.cc:343] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5f62b03db94d4ee3935866b5dd8bd53f
W20250625 07:52:21.339174 713 leader_election.cc:343] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.339349 713 leader_election.cc:304] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.339759 1617 raft_consensus.cc:2747] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:21.340327 1162 ts_tablet_manager.cc:1428] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 07:52:21.343302 1162 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:21.351822 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4da889781dbd41aebac86e9833cf2b7e consensus state has no leader
I20250625 07:52:21.352475 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 3758b4fc14df40f3bb9a6e77699981ea from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.352926 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_09566a49416a48758a3abe2ff7a69087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.353565 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09566a49416a48758a3abe2ff7a69087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.353482 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0e621427f14e48a497fcd6d20346cee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.354087 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e621427f14e48a497fcd6d20346cee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.355168 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289" } modify_external_catalogs: true
I20250625 07:52:21.355264 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 3eea1c257b584a029efa08922547b0cc (table default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab]). Expected version 2 got 0
I20250625 07:52:21.355355 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289" } modify_external_catalogs: true
I20250625 07:52:21.356014 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bebb528a00504a7e80d68cec36278f45" } modify_external_catalogs: true
I20250625 07:52:21.356175 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bebb528a00504a7e80d68cec36278f45" } modify_external_catalogs: true
W20250625 07:52:21.359495 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 670f3b7f0a904656af418d01ed6f58b4 consensus state has no leader
I20250625 07:52:21.360376 755 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.360845 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.360831 1582 raft_consensus.cc:491] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.361016 1582 raft_consensus.cc:513] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.361729 1582 leader_election.cc:290] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.362521 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:21.363211 851 leader_election.cc:343] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.363915 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:21.364825 1162 tablet_bootstrap.cc:654] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:21.366114 850 leader_election.cc:343] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.366250 850 leader_election.cc:304] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.366604 1582 raft_consensus.cc:2747] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:21.366624 1166 ts_tablet_manager.cc:1918] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.368643 1166 log.cc:1199] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/76fe692067954856908f936ced332c18
I20250625 07:52:21.369118 1166 ts_tablet_manager.cc:1939] T 76fe692067954856908f936ced332c18 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.369525 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e6e1b57ae284b1d9bc03891dc267c9c" } modify_external_catalogs: true
I20250625 07:52:21.369724 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e6e1b57ae284b1d9bc03891dc267c9c" } modify_external_catalogs: true
I20250625 07:52:21.370111 1166 tablet_replica.cc:331] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.370283 1166 raft_consensus.cc:2241] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.370476 1166 raft_consensus.cc:2270] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.371028 1166 ts_tablet_manager.cc:1905] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.371210 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 76fe692067954856908f936ced332c18 (table default.Table_33aedac05c3649c0a21c0367c4ff59a3 [id=b02af454e5f6454885578ace6c10ad97]) successfully deleted
W20250625 07:52:21.375588 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 44251.6us
I20250625 07:52:21.388568 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 5b04a575e0984f879f0179c8baa3f3fa from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.390055 590 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f815ed67e17347dcad804cbc1ba7660c
I20250625 07:52:21.391325 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_579b087c06ea4011b8e8445e7ea3cd0e" } modify_external_catalogs: true
I20250625 07:52:21.391588 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_579b087c06ea4011b8e8445e7ea3cd0e" } modify_external_catalogs: true
I20250625 07:52:21.392668 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9f09205826714483b879b30862cdf1c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.393474 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f09205826714483b879b30862cdf1c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.396085 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7 [id=3509d763da554384ba8b5ecefdbc5fab] alter complete (version 2)
I20250625 07:52:21.396750 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c632d0f66e464da088ba6a64e1d70e3b" } new_table_name: "default.Table_6f4572b322a446d594b17b6e16839583" modify_external_catalogs: true
I20250625 07:52:21.399466 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 35a93e9198d1417bb1b8e00560921483 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.400671 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.402035 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a" } modify_external_catalogs: true
I20250625 07:52:21.404346 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a" } modify_external_catalogs: true
W20250625 07:52:21.409722 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edda0c3f60f64ed0b965eca52c5b3591 consensus state has no leader
I20250625 07:52:21.411478 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_132ff7dc86c64138af1deb2266c2accc" } new_table_name: "default.Table_a3b92394c95f4ff5bb95019ea9de1300" modify_external_catalogs: true
W20250625 07:52:21.418496 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20750adfe4c14c55bddc2ea3bbf99ef2 has no consensus state
I20250625 07:52:21.421631 1163 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.421800 1163 ts_tablet_manager.cc:1397] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.005s
I20250625 07:52:21.422564 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ed4bfc88fee44f85afe168b7dcd1017e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.422880 1163 raft_consensus.cc:357] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.423086 1163 raft_consensus.cc:383] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.423179 1163 raft_consensus.cc:738] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.423437 1163 consensus_queue.cc:260] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:21.423866 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed4bfc88fee44f85afe168b7dcd1017e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.426555 1618 raft_consensus.cc:491] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.427397 1618 raft_consensus.cc:513] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.428318 1162 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:21.428484 1618 leader_election.cc:290] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.428681 1162 ts_tablet_manager.cc:1397] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.002s
I20250625 07:52:21.429164 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f63fdf2c8464c8fb58f412263d4e5c5" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:21.429482 1162 raft_consensus.cc:357] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:21.429680 713 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.429776 1162 raft_consensus.cc:383] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.429909 1162 raft_consensus.cc:738] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.430225 1162 consensus_queue.cc:260] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.430717 1163 ts_tablet_manager.cc:1428] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20250625 07:52:21.430336 713 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2f63fdf2c8464c8fb58f412263d4e5c5
I20250625 07:52:21.430917 713 leader_election.cc:304] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.431757 1162 ts_tablet_manager.cc:1428] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:21.431957 1617 raft_consensus.cc:2747] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:21.440037 1163 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.441659 1162 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:21.445647 1158 tablet_bootstrap.cc:492] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.445819 1158 ts_tablet_manager.cc:1397] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.164s user 0.018s sys 0.063s
I20250625 07:52:21.446655 1158 raft_consensus.cc:357] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:21.446882 1158 raft_consensus.cc:383] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.446990 1158 raft_consensus.cc:738] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.447253 1158 consensus_queue.cc:260] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:21.453882 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_da75eea0e9054b62ac5c228ce0fa5b27 [id=b0f32fb69f33447db7620299affa88c8]). Expected version 2 got 0
I20250625 07:52:21.455478 884 tablet_service.cc:1468] Processing CreateTablet for tablet d2c88638ac1843649fa7060cf1c62f90 (DEFAULT_TABLE table=default.Table_a37bdd5dd7c1405ea495ec1a66f47133 [id=864c4cd2a29d4e7e95d6eabe92d2ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.455724 1163 tablet_bootstrap.cc:654] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.455971 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c88638ac1843649fa7060cf1c62f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.456336 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:21.456867 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.457551 1160 tablet_replica.cc:331] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.457791 1160 raft_consensus.cc:2241] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.457958 1160 raft_consensus.cc:2270] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.457945 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d2c88638ac1843649fa7060cf1c62f90 (DEFAULT_TABLE table=default.Table_a37bdd5dd7c1405ea495ec1a66f47133 [id=864c4cd2a29d4e7e95d6eabe92d2ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.458302 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.458431 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c88638ac1843649fa7060cf1c62f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.460969 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.461130 1160 ts_tablet_manager.cc:1905] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.462762 755 tablet_service.cc:1468] Processing CreateTablet for tablet d2c88638ac1843649fa7060cf1c62f90 (DEFAULT_TABLE table=default.Table_a37bdd5dd7c1405ea495ec1a66f47133 [id=864c4cd2a29d4e7e95d6eabe92d2ce71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.463236 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2c88638ac1843649fa7060cf1c62f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.463753 1162 tablet_bootstrap.cc:654] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.469704 1166 ts_tablet_manager.cc:1918] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.469852 1166 log.cc:1199] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/5f62b03db94d4ee3935866b5dd8bd53f
I20250625 07:52:21.470217 1166 ts_tablet_manager.cc:1939] T 5f62b03db94d4ee3935866b5dd8bd53f P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.471276 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 5f62b03db94d4ee3935866b5dd8bd53f (table default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]) successfully deleted
I20250625 07:52:21.473387 1160 ts_tablet_manager.cc:1918] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.473613 1160 log.cc:1199] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:21.474349 1160 ts_tablet_manager.cc:1939] T 2d8d3300befc4f419070ae84725ee96a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.474538 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:21.475596 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7a53d8f50224460b932f4f44659336da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a53d8f50224460b932f4f44659336da already in progress: creating tablet
I20250625 07:52:21.476256 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2d8d3300befc4f419070ae84725ee96a (table default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc [id=f24c46ab73494d2b958f0fa750eadc63]) successfully deleted
I20250625 07:52:21.476581 1160 tablet_replica.cc:331] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.478114 1435 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 07:52:21.479686 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7863b48536e14b2a8d9173c8aa787fbf (DEFAULT_TABLE table=default.Table_c24353dd31c0489299840521478b82f4 [id=4d8f51c3793f46f997819971ac2dbf79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.480108 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7863b48536e14b2a8d9173c8aa787fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.480036 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7863b48536e14b2a8d9173c8aa787fbf (DEFAULT_TABLE table=default.Table_c24353dd31c0489299840521478b82f4 [id=4d8f51c3793f46f997819971ac2dbf79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.480383 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7863b48536e14b2a8d9173c8aa787fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.483512 753 tablet_service.cc:1468] Processing CreateTablet for tablet 51744ae4b84e47cea49fb54023777a41 (DEFAULT_TABLE table=default.Table_e53a5a947f2c4463a268d7c905ee326f [id=805872761a71403ea5c13d42edf3a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.483911 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51744ae4b84e47cea49fb54023777a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.486917 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 51744ae4b84e47cea49fb54023777a41 (DEFAULT_TABLE table=default.Table_e53a5a947f2c4463a268d7c905ee326f [id=805872761a71403ea5c13d42edf3a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.487197 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa (DEFAULT_TABLE table=default.Table_4208ed1cf6fc494f86365f0b22d2e390 [id=71146a99d1004fe2bab9185d896025f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.487287 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51744ae4b84e47cea49fb54023777a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.487545 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2481ca7e8db42a3a51270158bb2e2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.487807 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 539031fbe0674d018dbad8252ee5a6ed (DEFAULT_TABLE table=default.Table_f7740a0ca4b24e5899c7fec81bb78e8e [id=417b82cc406047618a034e91d55cb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.488157 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539031fbe0674d018dbad8252ee5a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.490249 746 tablet_service.cc:1468] Processing CreateTablet for tablet 539031fbe0674d018dbad8252ee5a6ed (DEFAULT_TABLE table=default.Table_f7740a0ca4b24e5899c7fec81bb78e8e [id=417b82cc406047618a034e91d55cb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.490612 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539031fbe0674d018dbad8252ee5a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.492978 742 tablet_service.cc:1468] Processing CreateTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa (DEFAULT_TABLE table=default.Table_4208ed1cf6fc494f86365f0b22d2e390 [id=71146a99d1004fe2bab9185d896025f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.493048 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e (DEFAULT_TABLE table=default.Table_579b087c06ea4011b8e8445e7ea3cd0e [id=505afe55ba5c46f4b02725881ccdd2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.493296 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2481ca7e8db42a3a51270158bb2e2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.493438 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.495216 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet ba24bbd0c4694930978a7750f07601d9 (table default.Table_f562abe21a314a6aaf1bf1574b9b908d [id=30de507c3fb344ecb071e9def7dcf119]). Expected version 1 got 0
I20250625 07:52:21.497005 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e (DEFAULT_TABLE table=default.Table_579b087c06ea4011b8e8445e7ea3cd0e [id=505afe55ba5c46f4b02725881ccdd2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.497371 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:21.501477 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet c8c015c202544914b74f1e245b496f82 (table default.Table_2b519d271b96497dafe0b274a8755066 [id=4a285d0804ea403bb60aa49bfde4bd9a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c8c015c202544914b74f1e245b496f82 consensus state has no leader
I20250625 07:52:21.478528 1163 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.507350 1163 ts_tablet_manager.cc:1397] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20250625 07:52:21.507727 1160 raft_consensus.cc:2241] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.508220 1163 raft_consensus.cc:357] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.481343 890 tablet_service.cc:1468] Processing CreateTablet for tablet 51744ae4b84e47cea49fb54023777a41 (DEFAULT_TABLE table=default.Table_e53a5a947f2c4463a268d7c905ee326f [id=805872761a71403ea5c13d42edf3a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.508935 1163 raft_consensus.cc:383] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.509032 1163 raft_consensus.cc:738] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.509380 1163 consensus_queue.cc:260] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.509892 1163 ts_tablet_manager.cc:1428] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:21.510280 1163 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.510360 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51744ae4b84e47cea49fb54023777a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.483691 889 tablet_service.cc:1468] Processing CreateTablet for tablet 539031fbe0674d018dbad8252ee5a6ed (DEFAULT_TABLE table=default.Table_f7740a0ca4b24e5899c7fec81bb78e8e [id=417b82cc406047618a034e91d55cb6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.484192 895 tablet_service.cc:1468] Processing CreateTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa (DEFAULT_TABLE table=default.Table_4208ed1cf6fc494f86365f0b22d2e390 [id=71146a99d1004fe2bab9185d896025f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.485486 880 tablet_service.cc:1468] Processing CreateTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e (DEFAULT_TABLE table=default.Table_579b087c06ea4011b8e8445e7ea3cd0e [id=505afe55ba5c46f4b02725881ccdd2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.517325 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539031fbe0674d018dbad8252ee5a6ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.519178 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2481ca7e8db42a3a51270158bb2e2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.519289 1163 tablet_bootstrap.cc:654] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.524986 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.527918 1166 tablet_replica.cc:331] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.528224 1166 raft_consensus.cc:2241] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.528384 1166 raft_consensus.cc:2270] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.528980 1166 ts_tablet_manager.cc:1905] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.531674 1158 ts_tablet_manager.cc:1428] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.086s user 0.001s sys 0.001s
I20250625 07:52:21.533932 1158 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.540190 1163 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.540403 1163 ts_tablet_manager.cc:1397] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.007s
I20250625 07:52:21.541433 1163 raft_consensus.cc:357] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.541626 1163 raft_consensus.cc:383] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.542801 1163 raft_consensus.cc:738] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.543578 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.543620 1163 consensus_queue.cc:260] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.544945 1163 ts_tablet_manager.cc:1428] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 07:52:21.545441 1163 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.545710 1158 tablet_bootstrap.cc:654] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.485874 884 tablet_service.cc:1468] Processing CreateTablet for tablet 7863b48536e14b2a8d9173c8aa787fbf (DEFAULT_TABLE table=default.Table_c24353dd31c0489299840521478b82f4 [id=4d8f51c3793f46f997819971ac2dbf79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.501451 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.549244 1166 ts_tablet_manager.cc:1918] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.549389 1166 log.cc:1199] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:21.549695 1166 ts_tablet_manager.cc:1939] T 9c770b41e0844786ad81c7b3579f9934 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.550747 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9c770b41e0844786ad81c7b3579f9934 (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:21.552770 1163 tablet_bootstrap.cc:654] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.553050 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7863b48536e14b2a8d9173c8aa787fbf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:21.561182 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5b19f6c181ee4e24883b0d7df822ab3a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b19f6c181ee4e24883b0d7df822ab3a already in progress: creating tablet
I20250625 07:52:21.561944 1160 raft_consensus.cc:2270] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.562672 1160 ts_tablet_manager.cc:1905] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:21.563638 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet cf175e17865e40ceb26aa955b91c2c13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cf175e17865e40ceb26aa955b91c2c13 already in progress: creating tablet
I20250625 07:52:21.570119 1163 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.570196 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:21.570305 1163 ts_tablet_manager.cc:1397] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20250625 07:52:21.571589 1163 raft_consensus.cc:357] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.571815 1163 raft_consensus.cc:383] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.571893 1163 raft_consensus.cc:738] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.572132 1163 consensus_queue.cc:260] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.572719 1163 ts_tablet_manager.cc:1428] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:21.573033 1163 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.572811 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3c414c85ddfa4e7dae5fdad4e25b06ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.574262 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e0319c343efd4139bb3d83e99569aa6c" } new_table_name: "default.Table_bcbc8715a04246c4a1760354c9e1f45d" modify_external_catalogs: true
W20250625 07:52:21.574376 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c414c85ddfa4e7dae5fdad4e25b06ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.574744 1163 tablet_bootstrap.cc:654] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.577056 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a53d8f50224460b932f4f44659336da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.577353 1161 tablet_replica.cc:331] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.577564 1161 raft_consensus.cc:2241] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.577726 1161 raft_consensus.cc:2270] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.578498 1161 ts_tablet_manager.cc:1905] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.581990 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.584709 1163 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.584911 1163 ts_tablet_manager.cc:1397] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250625 07:52:21.585518 1161 ts_tablet_manager.cc:1918] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.585747 1161 log.cc:1199] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7a53d8f50224460b932f4f44659336da
I20250625 07:52:21.585675 1163 raft_consensus.cc:357] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.585851 1163 raft_consensus.cc:383] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.585942 1163 raft_consensus.cc:738] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.586246 1161 ts_tablet_manager.cc:1939] T 7a53d8f50224460b932f4f44659336da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:21.586187 1163 consensus_queue.cc:260] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.586719 1163 ts_tablet_manager.cc:1428] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 07:52:21.588102 713 consensus_peers.cc:487] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:21.589440 1163 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.589375 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7a53d8f50224460b932f4f44659336da (table default.Table_2064fe78f7a74f389240146e3502eb6c [id=94cd213c64714f758795f9cbb1844e57]) successfully deleted
W20250625 07:52:21.591720 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 10c7f4b4187c428bbec136c85219da4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10c7f4b4187c428bbec136c85219da4d already in progress: creating tablet
I20250625 07:52:21.591969 1163 tablet_bootstrap.cc:654] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:21.592667 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 960ded22e2eb42d9a3494c020a4ea354 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 960ded22e2eb42d9a3494c020a4ea354 already in progress: creating tablet
I20250625 07:52:21.593600 1166 tablet_replica.cc:331] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.594328 1166 raft_consensus.cc:2241] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.595011 1166 raft_consensus.cc:2270] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.595988 1166 ts_tablet_manager.cc:1905] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.605105 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e
I20250625 07:52:21.605233 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 17be34eba92246df973f3d464e13fd30
I20250625 07:52:21.606034 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e1caebaa6b58419199c7477103213ec9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.606809 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1caebaa6b58419199c7477103213ec9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.607638 1163 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.607813 1163 ts_tablet_manager.cc:1397] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20250625 07:52:21.608198 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c35896915c304b1b8dada33cb186ca0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.608665 1163 raft_consensus.cc:357] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.608892 1163 raft_consensus.cc:383] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.608970 1163 raft_consensus.cc:738] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.609221 1163 consensus_queue.cc:260] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.609473 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.609486 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 17be34eba92246df973f3d464e13fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.609787 1163 ts_tablet_manager.cc:1428] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:21.609892 1617 consensus_queue.cc:1035] T 9cb2aa6dac1f4443be9782aac8f855af P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:21.610116 1163 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:21.610992 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c35896915c304b1b8dada33cb186ca0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.611904 1163 tablet_bootstrap.cc:654] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.612154 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.613649 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 17be34eba92246df973f3d464e13fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.614063 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.614378 890 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.614755 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:21.615155 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e already in progress: creating tablet
W20250625 07:52:21.615551 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 17be34eba92246df973f3d464e13fd30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17be34eba92246df973f3d464e13fd30 already in progress: creating tablet
I20250625 07:52:21.618085 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 35a93e9198d1417bb1b8e00560921483
I20250625 07:52:21.618458 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3758b4fc14df40f3bb9a6e77699981ea
I20250625 07:52:21.618659 591 catalog_manager.cc:6652] ReplaceTablet: tablet 35a93e9198d1417bb1b8e00560921483 deleted and replaced by tablet 99421eff752142348c8a899e7ac961b1
I20250625 07:52:21.618894 596 catalog_manager.cc:6652] ReplaceTablet: tablet 3758b4fc14df40f3bb9a6e77699981ea deleted and replaced by tablet 387a69222770408598184411225340e7
I20250625 07:52:21.619161 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d25d399f7faa4e56a98aeee2783040e5" } new_table_name: "default.Table_c250beeb533e41af826edcc10557bf9a" modify_external_catalogs: true
I20250625 07:52:21.619227 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_28849d2e25b0445483f83334953eedbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.619988 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28849d2e25b0445483f83334953eedbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.622210 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 17be34eba92246df973f3d464e13fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.622556 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a93e9198d1417bb1b8e00560921483 with delete_type TABLET_DATA_DELETED (replaced by tablet 99421eff752142348c8a899e7ac961b1) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.622799 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea with delete_type TABLET_DATA_DELETED (replaced by tablet 387a69222770408598184411225340e7) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.623323 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a93e9198d1417bb1b8e00560921483 with delete_type TABLET_DATA_DELETED (replaced by tablet 99421eff752142348c8a899e7ac961b1) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.623858 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea with delete_type TABLET_DATA_DELETED (replaced by tablet 387a69222770408598184411225340e7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.625361 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea with delete_type TABLET_DATA_DELETED (replaced by tablet 387a69222770408598184411225340e7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.625600 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a93e9198d1417bb1b8e00560921483 with delete_type TABLET_DATA_DELETED (replaced by tablet 99421eff752142348c8a899e7ac961b1) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:21.625874 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 35a93e9198d1417bb1b8e00560921483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a93e9198d1417bb1b8e00560921483 already in progress: creating tablet
W20250625 07:52:21.626168 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3758b4fc14df40f3bb9a6e77699981ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3758b4fc14df40f3bb9a6e77699981ea already in progress: creating tablet
I20250625 07:52:21.627513 1163 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.627588 751 tablet_service.cc:1468] Processing CreateTablet for tablet 69e2214bf6e54052bf8357b32caf17e6 (DEFAULT_TABLE table=default.Table_43525db1d2034f6f8460bbfccb691fc0 [id=5bbc1d3d38da4bd3ba037cef3b3782fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.627683 1163 ts_tablet_manager.cc:1397] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.006s
I20250625 07:52:21.628023 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e2214bf6e54052bf8357b32caf17e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.628582 1163 raft_consensus.cc:357] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:21.628786 1163 raft_consensus.cc:383] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.628872 1163 raft_consensus.cc:738] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.628895 753 tablet_service.cc:1468] Processing CreateTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.629133 1163 consensus_queue.cc:260] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:21.629722 1163 ts_tablet_manager.cc:1428] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:21.629980 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc85ec9546eb4da99b877c603d6b91e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.630009 1163 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.631796 1163 tablet_bootstrap.cc:654] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.640470 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 69e2214bf6e54052bf8357b32caf17e6 (DEFAULT_TABLE table=default.Table_43525db1d2034f6f8460bbfccb691fc0 [id=5bbc1d3d38da4bd3ba037cef3b3782fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.640937 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e2214bf6e54052bf8357b32caf17e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.640939 1162 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:21.641074 1162 ts_tablet_manager.cc:1397] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.200s user 0.008s sys 0.014s
I20250625 07:52:21.642300 1162 raft_consensus.cc:357] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.642470 1162 raft_consensus.cc:383] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.642552 1162 raft_consensus.cc:738] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.644198 889 tablet_service.cc:1468] Processing CreateTablet for tablet 69e2214bf6e54052bf8357b32caf17e6 (DEFAULT_TABLE table=default.Table_43525db1d2034f6f8460bbfccb691fc0 [id=5bbc1d3d38da4bd3ba037cef3b3782fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.644570 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e2214bf6e54052bf8357b32caf17e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.644951 1023 tablet_service.cc:1468] Processing CreateTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.645329 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc85ec9546eb4da99b877c603d6b91e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.646332 1166 ts_tablet_manager.cc:1918] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.646452 1166 log.cc:1199] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3c0edbf6cbdb49a6b3d3fff7f347c8ce
I20250625 07:52:21.646803 1166 ts_tablet_manager.cc:1939] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.647847 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (table default.Table_457927be3bfb45d1ab90fc5a3b8163d8 [id=9511cff29b2441689b47351734929304]) successfully deleted
I20250625 07:52:21.650341 1163 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.650547 1163 ts_tablet_manager.cc:1397] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.002s
I20250625 07:52:21.651429 1163 raft_consensus.cc:357] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:21.652204 1163 raft_consensus.cc:383] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.652321 1163 raft_consensus.cc:738] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.652601 1163 consensus_queue.cc:260] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:21.651401 746 tablet_service.cc:1468] Processing CreateTablet for tablet 80691f033eb44c6ba256b47acd868b70 (DEFAULT_TABLE table=default.Table_bd784c92cfd3446e8177e7bac2fa84fa [id=cb7da96031f64a119c4df0802e749b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.653162 1163 ts_tablet_manager.cc:1428] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:21.653281 755 tablet_service.cc:1468] Processing CreateTablet for tablet 03d999941e2a4265a64df7ec106a89a7 (DEFAULT_TABLE table=default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.653472 1163 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.653491 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80691f033eb44c6ba256b47acd868b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.653674 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d999941e2a4265a64df7ec106a89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.654557 888 tablet_service.cc:1468] Processing CreateTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.654943 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb03e70e9aa740f18d8cd0543f730f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.655074 886 tablet_service.cc:1468] Processing CreateTablet for tablet 383f8bc6a1cf485f9c56289fda15963f (DEFAULT_TABLE table=default.Table_696ed031c49440bc8b00f3c81a7d332a [id=18dd2783c90e43648e28f38c9e3f4f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.655434 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383f8bc6a1cf485f9c56289fda15963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.652069 742 tablet_service.cc:1468] Processing CreateTablet for tablet 184cd64146f141db8e870af121a7772f (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.657021 893 tablet_service.cc:1468] Processing CreateTablet for tablet 184cd64146f141db8e870af121a7772f (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.657205 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184cd64146f141db8e870af121a7772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.657405 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184cd64146f141db8e870af121a7772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.658102 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea (DEFAULT_TABLE table=default.Table_d25d399f7faa4e56a98aeee2783040e5 [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.658530 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.652621 756 tablet_service.cc:1468] Processing CreateTablet for tablet 383f8bc6a1cf485f9c56289fda15963f (DEFAULT_TABLE table=default.Table_696ed031c49440bc8b00f3c81a7d332a [id=18dd2783c90e43648e28f38c9e3f4f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.660976 1012 tablet_service.cc:1468] Processing CreateTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.661114 882 tablet_service.cc:1468] Processing CreateTablet for tablet 03d999941e2a4265a64df7ec106a89a7 (DEFAULT_TABLE table=default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.661190 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383f8bc6a1cf485f9c56289fda15963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.661340 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb03e70e9aa740f18d8cd0543f730f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.661510 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d999941e2a4265a64df7ec106a89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.662384 1163 tablet_bootstrap.cc:654] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.664321 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 383f8bc6a1cf485f9c56289fda15963f (DEFAULT_TABLE table=default.Table_696ed031c49440bc8b00f3c81a7d332a [id=18dd2783c90e43648e28f38c9e3f4f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.664733 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383f8bc6a1cf485f9c56289fda15963f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.664961 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 80691f033eb44c6ba256b47acd868b70 (DEFAULT_TABLE table=default.Table_bd784c92cfd3446e8177e7bac2fa84fa [id=cb7da96031f64a119c4df0802e749b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.665277 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80691f033eb44c6ba256b47acd868b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.669008 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 03d999941e2a4265a64df7ec106a89a7 (DEFAULT_TABLE table=default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.669337 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d999941e2a4265a64df7ec106a89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.670892 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 184cd64146f141db8e870af121a7772f (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.671257 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184cd64146f141db8e870af121a7772f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.650842 751 tablet_service.cc:1468] Processing CreateTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 (DEFAULT_TABLE table=default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.677148 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb03e70e9aa740f18d8cd0543f730f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.682379 753 tablet_service.cc:1468] Processing CreateTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea (DEFAULT_TABLE table=default.Table_d25d399f7faa4e56a98aeee2783040e5 [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.683105 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.688278 890 tablet_service.cc:1468] Processing CreateTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea (DEFAULT_TABLE table=default.Table_d25d399f7faa4e56a98aeee2783040e5 [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.688776 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.693022 895 tablet_service.cc:1468] Processing CreateTablet for tablet 80691f033eb44c6ba256b47acd868b70 (DEFAULT_TABLE table=default.Table_bd784c92cfd3446e8177e7bac2fa84fa [id=cb7da96031f64a119c4df0802e749b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.693043 1162 consensus_queue.cc:260] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.693593 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80691f033eb44c6ba256b47acd868b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.694384 1162 ts_tablet_manager.cc:1428] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20250625 07:52:21.695144 1162 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:21.697305 1162 tablet_bootstrap.cc:654] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.698212 880 tablet_service.cc:1468] Processing CreateTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.698760 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc85ec9546eb4da99b877c603d6b91e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.711447 1166 tablet_replica.cc:331] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.713521 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5b04a575e0984f879f0179c8baa3f3fa
I20250625 07:52:21.713637 1158 tablet_bootstrap.cc:492] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:21.713827 1158 ts_tablet_manager.cc:1397] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.180s user 0.002s sys 0.023s
I20250625 07:52:21.714164 592 catalog_manager.cc:6652] ReplaceTablet: tablet 5b04a575e0984f879f0179c8baa3f3fa deleted and replaced by tablet fed43adfae594e3b91ed1012838c3317
I20250625 07:52:21.714677 1158 raft_consensus.cc:357] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:21.714834 1158 raft_consensus.cc:383] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.714880 1158 raft_consensus.cc:738] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.715440 1158 consensus_queue.cc:260] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:21.715924 1158 ts_tablet_manager.cc:1428] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:21.716223 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:21.716554 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa with delete_type TABLET_DATA_DELETED (replaced by tablet fed43adfae594e3b91ed1012838c3317) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.716907 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa with delete_type TABLET_DATA_DELETED (replaced by tablet fed43adfae594e3b91ed1012838c3317) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.722761 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.725216 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.725276 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa with delete_type TABLET_DATA_DELETED (replaced by tablet fed43adfae594e3b91ed1012838c3317) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.725503 1166 raft_consensus.cc:2241] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.725693 1166 raft_consensus.cc:2270] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.726279 1166 ts_tablet_manager.cc:1905] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.727685 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet be7b755d95b14ecea7a691003455f381
I20250625 07:52:21.727782 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9f31104720614b028e373ae1500c31cc
I20250625 07:52:21.728641 1162 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:21.728855 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f31104720614b028e373ae1500c31cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.729156 593 master_service.cc:948] ReplaceTablet: received request to replace tablet d76681f162074b67a6b886887b5a3d61 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:21.729501 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.730049 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f31104720614b028e373ae1500c31cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.730064 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e06ecad6c7464ffe8ec6f632634307c9
I20250625 07:52:21.730636 594 catalog_manager.cc:6652] ReplaceTablet: tablet e06ecad6c7464ffe8ec6f632634307c9 deleted and replaced by tablet 328b21f669c442fb86259500e794d222
I20250625 07:52:21.730728 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 328b21f669c442fb86259500e794d222) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.731688 880 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22a46594f47deb8b142bc14c81a0b (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.732167 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22a46594f47deb8b142bc14c81a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.732836 884 tablet_service.cc:1515] Processing DeleteTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 328b21f669c442fb86259500e794d222) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:21.749936 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5b04a575e0984f879f0179c8baa3f3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b04a575e0984f879f0179c8baa3f3fa already in progress: creating tablet
W20250625 07:52:21.750356 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9f31104720614b028e373ae1500c31cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f31104720614b028e373ae1500c31cc already in progress: creating tablet
W20250625 07:52:21.750671 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet be7b755d95b14ecea7a691003455f381 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be7b755d95b14ecea7a691003455f381 already in progress: creating tablet
W20250625 07:52:21.750952 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e06ecad6c7464ffe8ec6f632634307c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e06ecad6c7464ffe8ec6f632634307c9 already in progress: creating tablet
I20250625 07:52:21.751649 890 tablet_service.cc:1468] Processing CreateTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 (DEFAULT_TABLE table=default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f [id=104e8b10958d4657b4ff37b593d5e8cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.751812 1162 ts_tablet_manager.cc:1397] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250625 07:52:21.752177 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a1d44dde9942a2bfd32a108b9dea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.750092 1027 tablet_service.cc:1468] Processing CreateTablet for tablet fd9677bf99f041c2ade65c94f0170609 (DEFAULT_TABLE table=default.Table_1b43beebf15d44d2a37382c52d1c35fb [id=28f257540e3d474588b3b5935970b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.750325 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 22f6c604ec1f456491ca197c7016d650 (DEFAULT_TABLE table=default.Table_0fb32606c55d45a7878009be37eb3ca8 [id=b89c197704ad408a8fabbf34576b19bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.754259 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9677bf99f041c2ade65c94f0170609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.757143 880 tablet_service.cc:1468] Processing CreateTablet for tablet fd9677bf99f041c2ade65c94f0170609 (DEFAULT_TABLE table=default.Table_1b43beebf15d44d2a37382c52d1c35fb [id=28f257540e3d474588b3b5935970b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.757558 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6c604ec1f456491ca197c7016d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.757707 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9677bf99f041c2ade65c94f0170609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.760313 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4aac2b881bfb4ada8db2169e25c54197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.750789 1028 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22a46594f47deb8b142bc14c81a0b (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:21.760891 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aac2b881bfb4ada8db2169e25c54197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.761149 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22a46594f47deb8b142bc14c81a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.761148 884 tablet_service.cc:1468] Processing CreateTablet for tablet 22f6c604ec1f456491ca197c7016d650 (DEFAULT_TABLE table=default.Table_0fb32606c55d45a7878009be37eb3ca8 [id=b89c197704ad408a8fabbf34576b19bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.761500 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6c604ec1f456491ca197c7016d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.762302 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e7ca753d58c84a17a3612fe36e374c7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.762794 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7ca753d58c84a17a3612fe36e374c7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.764191 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_66583e344f734087b51f1ea2962f4123"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.750572 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 (DEFAULT_TABLE table=default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f [id=104e8b10958d4657b4ff37b593d5e8cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:21.764624 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66583e344f734087b51f1ea2962f4123 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.764760 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a1d44dde9942a2bfd32a108b9dea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.765983 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4124f127e3e54adab2a8e666a1f18e48" } new_table_name: "default.Table_eb85d0d01edc400eabcdd8a8175dfbf9" modify_external_catalogs: true
W20250625 07:52:21.769104 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57c3d0810cf44a33b2acd50707931d26 consensus state has no leader
I20250625 07:52:21.775769 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:21.778887 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2c38b15750484886ad6bf5cc4253b510"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.779484 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c38b15750484886ad6bf5cc4253b510 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.780966 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_80cbf681213f4716b1b7fd920d71b59c" } modify_external_catalogs: true
I20250625 07:52:21.781206 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_80cbf681213f4716b1b7fd920d71b59c" } modify_external_catalogs: true
W20250625 07:52:21.782971 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1fc7cb613b6472ea34a0ea7bc8a08ea consensus state has no leader
I20250625 07:52:21.783452 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_825f0fc01f0a47eaa92ad842fe4e7c68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.784060 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_825f0fc01f0a47eaa92ad842fe4e7c68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.785881 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_701b07f0889e4b3a8765f597fa055673" } new_table_name: "default.Table_e9dd87383290477db1eb357d644d1b82" modify_external_catalogs: true
I20250625 07:52:21.793457 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.799088 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d76681f162074b67a6b886887b5a3d61
I20250625 07:52:21.799360 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e8fef495e82e483589a4868b7093361c" } new_table_name: "default.Table_fa7afe5cfd0647adb37b8be1a63705df" modify_external_catalogs: true
I20250625 07:52:21.799777 593 catalog_manager.cc:6652] ReplaceTablet: tablet d76681f162074b67a6b886887b5a3d61 deleted and replaced by tablet f1983df4902f48d2b408a99b440368e7
I20250625 07:52:21.800213 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_31fc7706ba984a4ebeb944ebfb3aeb25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.800913 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31fc7706ba984a4ebeb944ebfb3aeb25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:21.803095 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a07af928ff4d3d894b074ef0cf44eb consensus state has no leader
I20250625 07:52:21.803637 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.803628 880 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:21.804318 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d76681f162074b67a6b886887b5a3d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76681f162074b67a6b886887b5a3d61 already in progress: creating tablet
I20250625 07:52:21.804670 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c8c317c10e14332a3b3919808e4a512 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.733378 1158 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:21.804963 1161 tablet_replica.cc:331] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:21.805133 1161 raft_consensus.cc:2241] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.805313 1161 raft_consensus.cc:2270] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.806178 1161 ts_tablet_manager.cc:1905] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.806703 1158 tablet_bootstrap.cc:654] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.809527 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:21.810070 1162 raft_consensus.cc:357] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.810297 1162 raft_consensus.cc:383] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:21.810322 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffd98b2877b944f0bd484efa2a38a00a consensus state has no leader
I20250625 07:52:21.810371 1162 raft_consensus.cc:738] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.810595 1162 consensus_queue.cc:260] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.811051 1162 ts_tablet_manager.cc:1428] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20250625 07:52:21.811435 1162 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:21.814440 751 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.818099 1161 ts_tablet_manager.cc:1918] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.818255 1161 log.cc:1199] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3c8c317c10e14332a3b3919808e4a512
I20250625 07:52:21.818679 1161 ts_tablet_manager.cc:1939] T 3c8c317c10e14332a3b3919808e4a512 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:21.819036 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e910ce70a934be2be117a115fea3380" } modify_external_catalogs: true
I20250625 07:52:21.819235 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e910ce70a934be2be117a115fea3380" } modify_external_catalogs: true
I20250625 07:52:21.820859 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a976166b59b44d52a920b760df58965c" } modify_external_catalogs: true
I20250625 07:52:21.821053 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a976166b59b44d52a920b760df58965c" } modify_external_catalogs: true
I20250625 07:52:21.823112 1582 raft_consensus.cc:491] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.823278 1582 raft_consensus.cc:513] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.823891 1582 leader_election.cc:290] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.824488 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba24bbd0c4694930978a7750f07601d9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:21.825050 850 leader_election.cc:343] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.826139 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3c8c317c10e14332a3b3919808e4a512 (table default.Table_ce2222183ca64261b57847f5be67d81a [id=152fbc75a19b4eacb4909704af909443]) successfully deleted
I20250625 07:52:21.826388 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 328b21f669c442fb86259500e794d222) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.826504 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f31104720614b028e373ae1500c31cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.827028 751 tablet_service.cc:1468] Processing CreateTablet for tablet fd9677bf99f041c2ade65c94f0170609 (DEFAULT_TABLE table=default.Table_1b43beebf15d44d2a37382c52d1c35fb [id=28f257540e3d474588b3b5935970b3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.827447 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd9677bf99f041c2ade65c94f0170609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.829049 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba24bbd0c4694930978a7750f07601d9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:21.829488 746 tablet_service.cc:1468] Processing CreateTablet for tablet dfc22a46594f47deb8b142bc14c81a0b (DEFAULT_TABLE table=default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:21.830897 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1aa7fdb256b44e28a658e5e3edd4b97c has no consensus state
I20250625 07:52:21.832785 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfc22a46594f47deb8b142bc14c81a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.833693 753 tablet_service.cc:1468] Processing CreateTablet for tablet 22f6c604ec1f456491ca197c7016d650 (DEFAULT_TABLE table=default.Table_0fb32606c55d45a7878009be37eb3ca8 [id=b89c197704ad408a8fabbf34576b19bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.834041 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22f6c604ec1f456491ca197c7016d650. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.835680 755 tablet_service.cc:1468] Processing CreateTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 (DEFAULT_TABLE table=default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f [id=104e8b10958d4657b4ff37b593d5e8cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.836011 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91a1d44dde9942a2bfd32a108b9dea92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.836822 742 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:21.830315 851 leader_election.cc:343] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.838711 851 leader_election.cc:304] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:21.839308 1582 raft_consensus.cc:2747] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:21.826426 889 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.842231 1162 tablet_bootstrap.cc:654] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.846256 742 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.854027 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.825922 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:21.826196 884 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.865272 1582 raft_consensus.cc:491] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.865751 1582 raft_consensus.cc:513] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.866513 1582 leader_election.cc:290] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.872608 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 486463a8140c4e769cc79870d2acff78
I20250625 07:52:21.873592 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 486463a8140c4e769cc79870d2acff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.874601 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 486463a8140c4e769cc79870d2acff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.880155 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 (DEFAULT_TABLE table=default.Table_f5b40bffec5a46e1bee5f24f77334c4e [id=67548abe44834435af16fc09c19445ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.881012 1163 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:21.881111 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1f025775d04da09eda17af4ac4fe10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.881198 1163 ts_tablet_manager.cc:1397] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.228s user 0.086s sys 0.061s
I20250625 07:52:21.881994 1163 raft_consensus.cc:357] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.882175 1163 raft_consensus.cc:383] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:21.882248 1163 raft_consensus.cc:738] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:21.882467 1163 consensus_queue.cc:260] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.882952 1163 ts_tablet_manager.cc:1428] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:21.883229 1163 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:21.876912 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "651c74bf9a9b47d6ab5a875a5c304783" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:21.880532 751 tablet_service.cc:1468] Processing CreateTablet for tablet 71f459d573844d3a8abfd1b6e3813510 (DEFAULT_TABLE table=default.Table_2f2f17244c6943d8b807f3713710ad12 [id=354cf98c7fa94d3294235fd9c3488423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.885180 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f459d573844d3a8abfd1b6e3813510. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:21.886227 851 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.880019 746 tablet_service.cc:1468] Processing CreateTablet for tablet 62885e9218f64527854b79919e7725d5 (DEFAULT_TABLE table=default.Table_c363caab5c8c4e61b584674a8c476ac7 [id=98938ada8429491296620756c11bd84d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.887238 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62885e9218f64527854b79919e7725d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.887554 1163 tablet_bootstrap.cc:654] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:21.889465 888 tablet_service.cc:1468] Processing CreateTablet for tablet 64a5ff005b424ae5ae9831263abf58ef (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.889863 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a5ff005b424ae5ae9831263abf58ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.893342 895 tablet_service.cc:1468] Processing CreateTablet for tablet c1fc4392f58a4d91973506cb0939bb4c (DEFAULT_TABLE table=default.Table_ee0a0386d5404189ac77d4d731677aba [id=1e45bf32e3a048df950708e5d092028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.893733 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc4392f58a4d91973506cb0939bb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.876852 1166 ts_tablet_manager.cc:1918] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.870736 1160 ts_tablet_manager.cc:1918] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.894817 1160 log.cc:1199] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:21.895344 1160 ts_tablet_manager.cc:1939] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.896550 1160 tablet_replica.cc:331] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.896564 1166 log.cc:1199] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/ebb943df93284913b6a520e099399322
I20250625 07:52:21.896863 1160 raft_consensus.cc:2241] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.897079 1160 raft_consensus.cc:2270] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.897223 1166 ts_tablet_manager.cc:1939] T ebb943df93284913b6a520e099399322 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:21.897820 1160 ts_tablet_manager.cc:1905] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.898180 1166 tablet_replica.cc:331] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:21.898356 1166 raft_consensus.cc:2241] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.898991 1166 raft_consensus.cc:2270] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.900141 1166 ts_tablet_manager.cc:1905] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.902293 1582 raft_consensus.cc:491] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.902467 1582 raft_consensus.cc:513] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.903185 1582 leader_election.cc:290] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.906026 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1a5a40faa418420ea31bc1d46cab1f4a
I20250625 07:52:21.906895 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet ebb943df93284913b6a520e099399322 (table default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]) successfully deleted
I20250625 07:52:21.907234 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.907927 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5f917cfd2384a72920847f8b89358b9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:21.908612 851 leader_election.cc:343] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:21.913393 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (table default.Table_12188272bb8e4c26bb94db8721d37c26 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:21.913457 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:21.916285 890 tablet_service.cc:1468] Processing CreateTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 (DEFAULT_TABLE table=default.Table_c40382d89af6427eaf3c73a32b21d835 [id=8a65cb96796f40bb97ae671d0710b7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.917032 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a2929bad2340db9e6e0b5364ec8567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.919713 1160 ts_tablet_manager.cc:1918] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.926004 1160 log.cc:1199] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:21.926355 880 tablet_service.cc:1468] Processing CreateTablet for tablet 71f459d573844d3a8abfd1b6e3813510 (DEFAULT_TABLE table=default.Table_2f2f17244c6943d8b807f3713710ad12 [id=354cf98c7fa94d3294235fd9c3488423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.927042 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f459d573844d3a8abfd1b6e3813510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.926530 1160 ts_tablet_manager.cc:1939] T a3670cadfa2841949705c65c9b6c0927 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:21.933610 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9f2673ddf3e34a969d58c4514580bdcd" } modify_external_catalogs: true
I20250625 07:52:21.933869 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9f2673ddf3e34a969d58c4514580bdcd" } modify_external_catalogs: true
I20250625 07:52:21.934450 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_187c1ac9eb3649369d0c23cfd8eb18e8" } modify_external_catalogs: true
I20250625 07:52:21.934624 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_187c1ac9eb3649369d0c23cfd8eb18e8" } modify_external_catalogs: true
I20250625 07:52:21.934823 1160 tablet_replica.cc:331] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:21.935065 1160 raft_consensus.cc:2241] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:21.934964 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a3670cadfa2841949705c65c9b6c0927 (table default.Table_a881661cd92a4541a73b939c4532acc7 [id=dd2e80e6531a4e3f9be3b2fc19aec035]) successfully deleted
I20250625 07:52:21.935178 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 (DEFAULT_TABLE table=default.Table_f5b40bffec5a46e1bee5f24f77334c4e [id=67548abe44834435af16fc09c19445ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.935273 1160 raft_consensus.cc:2270] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:21.935583 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1f025775d04da09eda17af4ac4fe10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.935863 1160 ts_tablet_manager.cc:1905] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:21.937953 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_73f9bd402c6044089fc883b657511521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.938530 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73f9bd402c6044089fc883b657511521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.941833 884 tablet_service.cc:1468] Processing CreateTablet for tablet 62885e9218f64527854b79919e7725d5 (DEFAULT_TABLE table=default.Table_c363caab5c8c4e61b584674a8c476ac7 [id=98938ada8429491296620756c11bd84d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.947373 751 tablet_service.cc:1468] Processing CreateTablet for tablet c1fc4392f58a4d91973506cb0939bb4c (DEFAULT_TABLE table=default.Table_ee0a0386d5404189ac77d4d731677aba [id=1e45bf32e3a048df950708e5d092028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.947592 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62885e9218f64527854b79919e7725d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.947855 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc4392f58a4d91973506cb0939bb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.949097 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "651c74bf9a9b47d6ab5a875a5c304783" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:21.950687 755 tablet_service.cc:1468] Processing CreateTablet for tablet 64a5ff005b424ae5ae9831263abf58ef (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.951110 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a5ff005b424ae5ae9831263abf58ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.954643 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 (DEFAULT_TABLE table=default.Table_f5b40bffec5a46e1bee5f24f77334c4e [id=67548abe44834435af16fc09c19445ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.954842 753 tablet_service.cc:1468] Processing CreateTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 (DEFAULT_TABLE table=default.Table_c40382d89af6427eaf3c73a32b21d835 [id=8a65cb96796f40bb97ae671d0710b7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.957008 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:21.960124 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5de0c5bf60b24808af124c95833d9c25 (DEFAULT_TABLE table=default.Table_e7ca753d58c84a17a3612fe36e374c7a [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.961246 754 tablet_service.cc:1468] Processing CreateTablet for tablet 5ebb69eae9414c4ea940ee210526f79a (DEFAULT_TABLE table=default.Table_28849d2e25b0445483f83334953eedbe [id=55ffe92261ed4ab3b0152172f03cf12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.963102 1166 ts_tablet_manager.cc:1918] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.963250 1166 log.cc:1199] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2f63fdf2c8464c8fb58f412263d4e5c5
I20250625 07:52:21.965813 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5ebb69eae9414c4ea940ee210526f79a (DEFAULT_TABLE table=default.Table_28849d2e25b0445483f83334953eedbe [id=55ffe92261ed4ab3b0152172f03cf12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.966504 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebb69eae9414c4ea940ee210526f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.967950 886 tablet_service.cc:1468] Processing CreateTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 (DEFAULT_TABLE table=default.Table_09566a49416a48758a3abe2ff7a69087 [id=3f14380082174d9e9cd9df85c40d7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.968426 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f903943e9a4151b6fa1e71d1c05473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.968793 893 tablet_service.cc:1468] Processing CreateTablet for tablet 27a3f359857a47e799f0a5e3ca243828 (DEFAULT_TABLE table=default.Table_3c414c85ddfa4e7dae5fdad4e25b06ac [id=a8013169e5984229977a27a57f9e672e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.969151 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a3f359857a47e799f0a5e3ca243828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.972561 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_402d22d187b34d4caa19848d9bcc1612" } new_table_name: "default.Table_fffd92726b724fd09763acbb604f0941" modify_external_catalogs: true
I20250625 07:52:21.973450 883 tablet_service.cc:1468] Processing CreateTablet for tablet 5d019a5abdbe452d98f3d30ce45417c0 (DEFAULT_TABLE table=default.Table_0e621427f14e48a497fcd6d20346cee7 [id=00d1ab8388b24ed4bf735cac083a51e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.973901 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d019a5abdbe452d98f3d30ce45417c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.981667 1582 raft_consensus.cc:491] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:21.981853 1582 raft_consensus.cc:513] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:21.982550 1582 leader_election.cc:290] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:21.982841 885 tablet_service.cc:1468] Processing CreateTablet for tablet c847e0f7d9cc4199913e73ef2f4302fd (DEFAULT_TABLE table=default.Table_3921e8fba62d419b997de9a801f2274a [id=cf9a7939b4fb450089a76437005094fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.983371 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c847e0f7d9cc4199913e73ef2f4302fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.987819 1160 ts_tablet_manager.cc:1918] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:21.988421 1160 log.cc:1199] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:21.991288 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e925686f85164a9b94c4c280ebebfadb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.991981 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e925686f85164a9b94c4c280ebebfadb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.992961 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a85de53a06a54f77a07282be4323dd74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:21.993317 892 tablet_service.cc:1468] Processing CreateTablet for tablet a8f25febb41c403ca8be268206157756 (DEFAULT_TABLE table=default.Table_ed4bfc88fee44f85afe168b7dcd1017e [id=1a0718ad3cc64932b9fe0b972c4acdbf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:21.993484 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a85de53a06a54f77a07282be4323dd74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.993393 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa7fdb256b44e28a658e5e3edd4b97c (DEFAULT_TABLE table=default.Table_fa7afe5cfd0647adb37b8be1a63705df [id=85c542fc6f0f43c2b5d2aa0f0007ec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.994330 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8f25febb41c403ca8be268206157756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.994664 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa7fdb256b44e28a658e5e3edd4b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.995270 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6288489a16da45cf8a64efae2aa3f532" } new_table_name: "default.Table_0aff505ed58e44ffaaea42615be646d1" modify_external_catalogs: true
I20250625 07:52:21.995539 894 tablet_service.cc:1468] Processing CreateTablet for tablet 387a69222770408598184411225340e7 (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.996042 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 387a69222770408598184411225340e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.996750 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_94af0d30f3b8498c8bd98dbeec6e5085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:21.997242 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94af0d30f3b8498c8bd98dbeec6e5085 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:21.997535 881 tablet_service.cc:1468] Processing CreateTablet for tablet 99421eff752142348c8a899e7ac961b1 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.997937 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99421eff752142348c8a899e7ac961b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:21.999348 887 tablet_service.cc:1468] Processing CreateTablet for tablet 328b21f669c442fb86259500e794d222 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:21.999717 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b21f669c442fb86259500e794d222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.000938 891 tablet_service.cc:1468] Processing CreateTablet for tablet fed43adfae594e3b91ed1012838c3317 (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.001260 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed43adfae594e3b91ed1012838c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.006561 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ed4947df987b4ce48a2209dea3bf6dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.007216 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed4947df987b4ce48a2209dea3bf6dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.010255 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_53313429425a455cb97296e7d4b6f0dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.010946 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53313429425a455cb97296e7d4b6f0dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.011853 1582 raft_consensus.cc:491] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.012285 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_46be1cd72d9843de91659b93152efdd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.012792 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46be1cd72d9843de91659b93152efdd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.017105 1160 ts_tablet_manager.cc:1939] T 7ad78944352e4274acd8d9889b1d32b8 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.020421 1160 tablet_replica.cc:331] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.038425 1582 raft_consensus.cc:513] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.039433 1582 leader_election.cc:290] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.039918 1582 raft_consensus.cc:491] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.040105 1582 raft_consensus.cc:513] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.041141 1582 leader_election.cc:290] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.043350 1160 raft_consensus.cc:2241] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.043823 1160 raft_consensus.cc:2270] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.044832 1160 ts_tablet_manager.cc:1905] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.051723 1582 raft_consensus.cc:491] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.052013 1582 raft_consensus.cc:513] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.059202 888 tablet_service.cc:1468] Processing CreateTablet for tablet 05c040937f7646f5a0fbb98d533250b8 (DEFAULT_TABLE table=default.Table_31fc7706ba984a4ebeb944ebfb3aeb25 [id=c0e0b880ad4e4ecc9041e85a5acc5fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.077215 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c040937f7646f5a0fbb98d533250b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.066992 1160 ts_tablet_manager.cc:1918] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.082849 1160 log.cc:1199] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7f196473315441c6bf8b5173ed46f833
I20250625 07:52:22.083297 1160 ts_tablet_manager.cc:1939] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.083936 1160 tablet_replica.cc:331] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.084157 1160 raft_consensus.cc:2241] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.084350 1160 raft_consensus.cc:2270] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.084975 1160 ts_tablet_manager.cc:1905] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.072959 1582 leader_election.cc:290] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.089394 893 tablet_service.cc:1468] Processing CreateTablet for tablet a0342575a2b34bf4b2390968e6acbe5a (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.089936 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0342575a2b34bf4b2390968e6acbe5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.081321 895 tablet_service.cc:1468] Processing CreateTablet for tablet 5de0c5bf60b24808af124c95833d9c25 (DEFAULT_TABLE table=default.Table_e7ca753d58c84a17a3612fe36e374c7a [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.093704 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de0c5bf60b24808af124c95833d9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.082490 894 tablet_service.cc:1468] Processing CreateTablet for tablet 3d2f251602454a1eadf4e2803e226531 (DEFAULT_TABLE table=default.Table_825f0fc01f0a47eaa92ad842fe4e7c68 [id=ac1f155584c44b8491d7997f6ad348c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.061075 882 tablet_service.cc:1468] Processing CreateTablet for tablet f6fbe5526772435eb6ff141715994b65 (DEFAULT_TABLE table=default.Table_9ede02c06bf540edbff83b6d2e68d206 [id=767b2a30b40a475db021cb7e06852cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.094691 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d2f251602454a1eadf4e2803e226531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.101363 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fbe5526772435eb6ff141715994b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.101698 885 tablet_service.cc:1468] Processing CreateTablet for tablet 0f9955688a574c80a8b14c2018ed41f7 (DEFAULT_TABLE table=default.Table_e342d843577548959d8be6a80119e769 [id=fd849b1f759a49f6aa52e840c309cbb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.102196 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9955688a574c80a8b14c2018ed41f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.092936 1160 ts_tablet_manager.cc:1918] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.107084 1160 log.cc:1199] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a7cb2bcdd07c42e680eb4f155212423f
I20250625 07:52:22.107565 1160 ts_tablet_manager.cc:1939] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.112768 1160 tablet_replica.cc:331] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.113054 1160 raft_consensus.cc:2241] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.115929 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7ad78944352e4274acd8d9889b1d32b8 (table default.Table_bb291540701c4a4fbb5f3fa31fc4b684 [id=d24d9520ff924056a204dfd910f4dbef]) successfully deleted
I20250625 07:52:22.116515 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7f196473315441c6bf8b5173ed46f833 (table default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]) successfully deleted
I20250625 07:52:22.117203 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a7cb2bcdd07c42e680eb4f155212423f (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
I20250625 07:52:22.114951 878 tablet_service.cc:1468] Processing CreateTablet for tablet aba6420802914efe85111282398e6270 (DEFAULT_TABLE table=default.Table_40088cb2328f4d629da1e5d5341d135d [id=c836336c95794b4d8e4a4829545b3528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.115335 877 tablet_service.cc:1468] Processing CreateTablet for tablet e8750a4e01f449f3abcb0a283da1d9db (DEFAULT_TABLE table=default.Table_66583e344f734087b51f1ea2962f4123 [id=fd09e9ddd8e9468ba7df7083942e1b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.115554 876 tablet_service.cc:1468] Processing CreateTablet for tablet d43d98c0c51b40c6bdb0c2705465038e (DEFAULT_TABLE table=default.Table_6a808c6d46a94091b18c21073a499b8a [id=b6489a827b56434380514e9565a4aacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.116295 894 tablet_service.cc:1468] Processing CreateTablet for tablet 20750adfe4c14c55bddc2ea3bbf99ef2 (DEFAULT_TABLE table=default.Table_a3b92394c95f4ff5bb95019ea9de1300 [id=3a7146572e604811af6284d71852ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.118218 1160 raft_consensus.cc:2270] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.118956 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1f025775d04da09eda17af4ac4fe10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.122336 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:22.123248 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5ad0a090d681485683aa607f6f29b2b1" } new_table_name: "default.Table_42c59eeeff4e404d8aa4c201145576b8" modify_external_catalogs: true
I20250625 07:52:22.126647 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 44bfcc2b20ba4d0b83641988f2c0ee35
I20250625 07:52:22.127430 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1f83aa0aa3e8436281b7c9886cdc096c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:22.127638 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 62885e9218f64527854b79919e7725d5 (DEFAULT_TABLE table=default.Table_c363caab5c8c4e61b584674a8c476ac7 [id=98938ada8429491296620756c11bd84d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.128005 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f83aa0aa3e8436281b7c9886cdc096c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.129535 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62885e9218f64527854b79919e7725d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.129554 1160 ts_tablet_manager.cc:1905] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:22.131590 850 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.131764 850 leader_election.cc:304] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.132310 850 raft_consensus.cc:2684] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:22.132504 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5f917cfd2384a72920847f8b89358b9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.133766 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba6420802914efe85111282398e6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.134428 1485 raft_consensus.cc:491] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.135879 1485 raft_consensus.cc:513] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.136606 1485 leader_election.cc:290] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:22.136914 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 64a5ff005b424ae5ae9831263abf58ef (DEFAULT_TABLE table=default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.137363 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 486463a8140c4e769cc79870d2acff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.137847 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a5ff005b424ae5ae9831263abf58ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.116763 892 tablet_service.cc:1468] Processing CreateTablet for tablet 07837660a7db47299b8e11f3335cb5d7 (DEFAULT_TABLE table=default.Table_3666ef79a51b4d33a802675181e67fac [id=41cb67813536436fba87f4c8a020e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.114925 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba (DEFAULT_TABLE table=default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32 [id=a3bf600f08114191807e33fa9f466cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.139362 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 (DEFAULT_TABLE table=default.Table_c40382d89af6427eaf3c73a32b21d835 [id=8a65cb96796f40bb97ae671d0710b7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.139721 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 71f459d573844d3a8abfd1b6e3813510 (DEFAULT_TABLE table=default.Table_2f2f17244c6943d8b807f3713710ad12 [id=354cf98c7fa94d3294235fd9c3488423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.143947 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_abc25ebdf79c4455be53d3dfb090fdd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:22.144316 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a2929bad2340db9e6e0b5364ec8567. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.144572 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abc25ebdf79c4455be53d3dfb090fdd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.145917 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_35586b691c9a48068784a8147f80b6ef" } modify_external_catalogs: true
I20250625 07:52:22.146112 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_35586b691c9a48068784a8147f80b6ef" } modify_external_catalogs: true
W20250625 07:52:22.146450 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb23f900faf6421fb6e72f2b5202d0fd consensus state has no leader
W20250625 07:52:22.146854 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e109c9dca7444681ac517c7fc3e14dfa consensus state has no leader
I20250625 07:52:22.146979 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1386b0806db34e7482d4483c86eed121" } new_table_name: "default.Table_4f5e9909515b4b28b92b7ece88d91b4b" modify_external_catalogs: true
I20250625 07:52:22.147267 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.147806 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.150218 1029 tablet_service.cc:1468] Processing CreateTablet for tablet c1fc4392f58a4d91973506cb0939bb4c (DEFAULT_TABLE table=default.Table_ee0a0386d5404189ac77d4d731677aba [id=1e45bf32e3a048df950708e5d092028c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.151676 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71f459d573844d3a8abfd1b6e3813510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.157821 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 05c040937f7646f5a0fbb98d533250b8 (DEFAULT_TABLE table=default.Table_31fc7706ba984a4ebeb944ebfb3aeb25 [id=c0e0b880ad4e4ecc9041e85a5acc5fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.158210 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5fbde73edcc941ebb33753389aea2dc8" } modify_external_catalogs: true
I20250625 07:52:22.158454 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5fbde73edcc941ebb33753389aea2dc8" } modify_external_catalogs: true
I20250625 07:52:22.159147 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bd784c92cfd3446e8177e7bac2fa84fa" } new_table_name: "default.Table_d77337df3beb48698fd178bc5b2783d2" modify_external_catalogs: true
I20250625 07:52:22.160880 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_28849d2e25b0445483f83334953eedbe" } modify_external_catalogs: true
I20250625 07:52:22.161072 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_28849d2e25b0445483f83334953eedbe" } modify_external_catalogs: true
I20250625 07:52:22.163633 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_06a0b466c0dc496badbf8a1b18535c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.164292 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06a0b466c0dc496badbf8a1b18535c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.165506 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1fc4392f58a4d91973506cb0939bb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.173436 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu with pid 1268
W20250625 07:52:22.177606 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3823279d7e06435484f1b82836ebcf39 consensus state has no leader
I20250625 07:52:22.178952 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5ebb69eae9414c4ea940ee210526f79a (DEFAULT_TABLE table=default.Table_28849d2e25b0445483f83334953eedbe [id=55ffe92261ed4ab3b0152172f03cf12d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.179409 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f196473315441c6bf8b5173ed46f833" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.179821 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 5de0c5bf60b24808af124c95833d9c25 (DEFAULT_TABLE table=default.Table_e7ca753d58c84a17a3612fe36e374c7a [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.180832 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43d98c0c51b40c6bdb0c2705465038e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.186038 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32a2929bad2340db9e6e0b5364ec8567. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.188308 1166 ts_tablet_manager.cc:1939] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:22.189380 1166 tablet_replica.cc:331] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:22.192271 1617 raft_consensus.cc:491] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.192476 1617 raft_consensus.cc:513] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.190599 1582 raft_consensus.cc:491] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.193295 1617 leader_election.cc:290] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.193892 750 tablet_service.cc:1468] Processing CreateTablet for tablet 27a3f359857a47e799f0a5e3ca243828 (DEFAULT_TABLE table=default.Table_3c414c85ddfa4e7dae5fdad4e25b06ac [id=a8013169e5984229977a27a57f9e672e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.194306 742 tablet_service.cc:1468] Processing CreateTablet for tablet 05c040937f7646f5a0fbb98d533250b8 (DEFAULT_TABLE table=default.Table_31fc7706ba984a4ebeb944ebfb3aeb25 [id=c0e0b880ad4e4ecc9041e85a5acc5fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.194800 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20750adfe4c14c55bddc2ea3bbf99ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.198506 895 tablet_service.cc:1468] Processing CreateTablet for tablet f1983df4902f48d2b408a99b440368e7 (DEFAULT_TABLE table=default.Table_5d201449036c4c1a8351c46d645de96c [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.199020 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445bfed532004108a12c8fd78370c1cb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.199108 883 tablet_service.cc:1468] Processing CreateTablet for tablet 51d1ab1dc05c4758acba0f3f436a3e6f (DEFAULT_TABLE table=default.Table_c35896915c304b1b8dada33cb186ca0e [id=42ab65ca881947ccac05139c8a5e9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.199155 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445bfed532004108a12c8fd78370c1cb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.200096 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.200467 743 tablet_service.cc:1468] Processing CreateTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 (DEFAULT_TABLE table=default.Table_09566a49416a48758a3abe2ff7a69087 [id=3f14380082174d9e9cd9df85c40d7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.202026 882 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.202737 1582 raft_consensus.cc:513] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.203156 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 27a3f359857a47e799f0a5e3ca243828 (DEFAULT_TABLE table=default.Table_3c414c85ddfa4e7dae5fdad4e25b06ac [id=a8013169e5984229977a27a57f9e672e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.202421 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.203756 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3b6dc279984d7fba83a6308fbc0fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.203958 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5d019a5abdbe452d98f3d30ce45417c0 (DEFAULT_TABLE table=default.Table_0e621427f14e48a497fcd6d20346cee7 [id=00d1ab8388b24ed4bf735cac083a51e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.204913 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa7fdb256b44e28a658e5e3edd4b97c (DEFAULT_TABLE table=default.Table_fa7afe5cfd0647adb37b8be1a63705df [id=85c542fc6f0f43c2b5d2aa0f0007ec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.205601 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 387a69222770408598184411225340e7 (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.205674 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.206162 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21658216da3544a2b8b8f39667782ffb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.206332 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 99421eff752142348c8a899e7ac961b1 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.207162 1166 raft_consensus.cc:2241] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.207234 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8750a4e01f449f3abcb0a283da1d9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.207408 1166 raft_consensus.cc:2270] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.208137 1166 ts_tablet_manager.cc:1905] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.208312 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f196473315441c6bf8b5173ed46f833" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:22.208541 851 leader_election.cc:343] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.208644 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7cb2bcdd07c42e680eb4f155212423f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.204656 1020 tablet_service.cc:1468] Processing CreateTablet for tablet f6fbe5526772435eb6ff141715994b65 (DEFAULT_TABLE table=default.Table_9ede02c06bf540edbff83b6d2e68d206 [id=767b2a30b40a475db021cb7e06852cf0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.210059 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ccddd630a82341cba7a857e67a349fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccddd630a82341cba7a857e67a349fe0 already in progress: creating tablet
I20250625 07:52:22.214100 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebb69eae9414c4ea940ee210526f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.215037 881 tablet_service.cc:1468] Processing CreateTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 (DEFAULT_TABLE table=default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5 [id=6f669706429b46738a022387976f26e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.220136 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1983df4902f48d2b408a99b440368e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.221653 850 leader_election.cc:343] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.222721 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7cb2bcdd07c42e680eb4f155212423f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.223408 885 tablet_service.cc:1468] Processing CreateTablet for tablet a5a5ca65ab3e4f759b0b972aa793732e (DEFAULT_TABLE table=default.Table_e1caebaa6b58419199c7477103213ec9 [id=f99197b69ce0473da6969318c8218094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.225632 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9b83f4c8544946d9adb3346df200b215 (DEFAULT_TABLE table=default.Table_73f9bd402c6044089fc883b657511521 [id=9ae1c8331e2149b3a32cf69a11f834c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.234726 886 tablet_service.cc:1468] Processing CreateTablet for tablet b1cdde72919a4a3ca29434d475c2d44f (DEFAULT_TABLE table=default.Table_9f09205826714483b879b30862cdf1c9 [id=fe703b8887274f0a86349b7c717a9d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.236399 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de0c5bf60b24808af124c95833d9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.239310 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ebb69eae9414c4ea940ee210526f79a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.243357 1582 leader_election.cc:290] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.243577 1582 raft_consensus.cc:491] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.243510 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.244266 1582 raft_consensus.cc:513] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.244946 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f903943e9a4151b6fa1e71d1c05473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.245673 1582 leader_election.cc:290] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.245877 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5de1492e483e47919ee5581ecee227be" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:22.246522 893 tablet_service.cc:1468] Processing CreateTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.247175 850 leader_election.cc:304] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.248785 850 leader_election.cc:343] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.250958 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21658216da3544a2b8b8f39667782ffb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.252882 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a3f359857a47e799f0a5e3ca243828. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.252862 851 leader_election.cc:343] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:22.253137 851 leader_election.cc:343] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.253648 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c040937f7646f5a0fbb98d533250b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.253839 851 leader_election.cc:343] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.253962 851 leader_election.cc:304] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.254806 851 raft_consensus.cc:2684] T 7f196473315441c6bf8b5173ed46f833 P 429026e8d98d46339db143f84897a092: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:22.256705 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 486463a8140c4e769cc79870d2acff78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 486463a8140c4e769cc79870d2acff78 already in progress: creating tablet
W20250625 07:52:22.255962 851 leader_election.cc:343] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.258040 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31942f0998bb4118a91f476325851d5c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.257083 1582 raft_consensus.cc:2747] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.255139 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07837660a7db47299b8e11f3335cb5d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.261777 851 leader_election.cc:343] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:22.256043 850 leader_election.cc:343] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.262007 850 leader_election.cc:304] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.263183 850 leader_election.cc:343] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.263319 850 leader_election.cc:304] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.263870 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31942f0998bb4118a91f476325851d5c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.264238 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b83f4c8544946d9adb3346df200b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.264523 985 leader_election.cc:304] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:22.265782 1582 raft_consensus.cc:2747] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.267766 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cdde72919a4a3ca29434d475c2d44f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.268838 850 raft_consensus.cc:2684] T a7cb2bcdd07c42e680eb4f155212423f P 429026e8d98d46339db143f84897a092: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:22.269611 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d1ab1dc05c4758acba0f3f436a3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.269996 1485 raft_consensus.cc:2747] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.270579 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a5ca65ab3e4f759b0b972aa793732e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.271596 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efffbfca58d44ffa97e56b5c10e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.272318 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73561bac3bf4ece9da5433ab4d19b4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.274363 850 leader_election.cc:343] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.274756 850 leader_election.cc:304] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.275230 850 leader_election.cc:343] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.288823 850 leader_election.cc:304] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.289170 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 (DEFAULT_TABLE table=default.Table_09566a49416a48758a3abe2ff7a69087 [id=3f14380082174d9e9cd9df85c40d7dc5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.289387 850 leader_election.cc:343] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.289464 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c847e0f7d9cc4199913e73ef2f4302fd (DEFAULT_TABLE table=default.Table_3921e8fba62d419b997de9a801f2274a [id=cf9a7939b4fb450089a76437005094fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.289680 1026 tablet_service.cc:1468] Processing CreateTablet for tablet a8f25febb41c403ca8be268206157756 (DEFAULT_TABLE table=default.Table_ed4bfc88fee44f85afe168b7dcd1017e [id=1a0718ad3cc64932b9fe0b972c4acdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.289881 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 328b21f669c442fb86259500e794d222 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.289973 850 leader_election.cc:304] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.290074 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fbe5526772435eb6ff141715994b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.290637 1582 raft_consensus.cc:2747] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.290838 1582 raft_consensus.cc:2747] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.291528 1582 raft_consensus.cc:2747] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.293596 1162 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:22.294121 1162 ts_tablet_manager.cc:1397] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.483s user 0.064s sys 0.106s
I20250625 07:52:22.295320 1162 raft_consensus.cc:357] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.295720 1162 raft_consensus.cc:383] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.295890 1162 raft_consensus.cc:738] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.296119 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d019a5abdbe452d98f3d30ce45417c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.296304 1162 consensus_queue.cc:260] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.297602 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa7fdb256b44e28a658e5e3edd4b97c (DEFAULT_TABLE table=default.Table_fa7afe5cfd0647adb37b8be1a63705df [id=85c542fc6f0f43c2b5d2aa0f0007ec02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.298175 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa7fdb256b44e28a658e5e3edd4b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.300897 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 387a69222770408598184411225340e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.304886 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99421eff752142348c8a899e7ac961b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.305451 744 tablet_service.cc:1468] Processing CreateTablet for tablet a8f25febb41c403ca8be268206157756 (DEFAULT_TABLE table=default.Table_ed4bfc88fee44f85afe168b7dcd1017e [id=1a0718ad3cc64932b9fe0b972c4acdbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.305981 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8f25febb41c403ca8be268206157756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.309361 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27a3f359857a47e799f0a5e3ca243828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.312892 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa7fdb256b44e28a658e5e3edd4b97c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.313444 747 tablet_service.cc:1468] Processing CreateTablet for tablet 387a69222770408598184411225340e7 (DEFAULT_TABLE table=default.Table_091524b607f045b985a03d229bc7c8e4 [id=482c482c407347039b6e726c1ce95a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.313994 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 387a69222770408598184411225340e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.317063 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05c040937f7646f5a0fbb98d533250b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.321465 757 tablet_service.cc:1468] Processing CreateTablet for tablet 99421eff752142348c8a899e7ac961b1 (DEFAULT_TABLE table=default.Table_1386b0806db34e7482d4483c86eed121 [id=bf668cc9e9db4f529d2badcae5254439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.322046 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99421eff752142348c8a899e7ac961b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.324827 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.325325 1018 tablet_service.cc:1468] Processing CreateTablet for tablet fed43adfae594e3b91ed1012838c3317 (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.327791 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_26efe610b5ee4ac1b55ac4d7d91c7040" } modify_external_catalogs: true
I20250625 07:52:22.328008 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_26efe610b5ee4ac1b55ac4d7d91c7040" } modify_external_catalogs: true
I20250625 07:52:22.333518 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40088cb2328f4d629da1e5d5341d135d" } modify_external_catalogs: true
I20250625 07:52:22.333743 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40088cb2328f4d629da1e5d5341d135d" } modify_external_catalogs: true
I20250625 07:52:22.338963 751 tablet_service.cc:1468] Processing CreateTablet for tablet c847e0f7d9cc4199913e73ef2f4302fd (DEFAULT_TABLE table=default.Table_3921e8fba62d419b997de9a801f2274a [id=cf9a7939b4fb450089a76437005094fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.339433 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c847e0f7d9cc4199913e73ef2f4302fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.339457 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 50091.8us
I20250625 07:52:22.345064 755 tablet_service.cc:1468] Processing CreateTablet for tablet f6fbe5526772435eb6ff141715994b65 (DEFAULT_TABLE table=default.Table_9ede02c06bf540edbff83b6d2e68d206 [id=767b2a30b40a475db021cb7e06852cf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.345513 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fbe5526772435eb6ff141715994b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.353389 750 tablet_service.cc:1468] Processing CreateTablet for tablet 51d1ab1dc05c4758acba0f3f436a3e6f (DEFAULT_TABLE table=default.Table_c35896915c304b1b8dada33cb186ca0e [id=42ab65ca881947ccac05139c8a5e9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.353752 757 tablet_service.cc:1468] Processing CreateTablet for tablet a0342575a2b34bf4b2390968e6acbe5a (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.353853 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d1ab1dc05c4758acba0f3f436a3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.354074 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0342575a2b34bf4b2390968e6acbe5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.354188 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0f9955688a574c80a8b14c2018ed41f7 (DEFAULT_TABLE table=default.Table_e342d843577548959d8be6a80119e769 [id=fd849b1f759a49f6aa52e840c309cbb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.354521 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9955688a574c80a8b14c2018ed41f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.357025 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 (DEFAULT_TABLE table=default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5 [id=6f669706429b46738a022387976f26e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.357432 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efffbfca58d44ffa97e56b5c10e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.365047 744 tablet_service.cc:1468] Processing CreateTablet for tablet b1cdde72919a4a3ca29434d475c2d44f (DEFAULT_TABLE table=default.Table_9f09205826714483b879b30862cdf1c9 [id=fe703b8887274f0a86349b7c717a9d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.365496 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cdde72919a4a3ca29434d475c2d44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.369714 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b1cdde72919a4a3ca29434d475c2d44f (DEFAULT_TABLE table=default.Table_9f09205826714483b879b30862cdf1c9 [id=fe703b8887274f0a86349b7c717a9d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.372934 747 tablet_service.cc:1468] Processing CreateTablet for tablet 3d2f251602454a1eadf4e2803e226531 (DEFAULT_TABLE table=default.Table_825f0fc01f0a47eaa92ad842fe4e7c68 [id=ac1f155584c44b8491d7997f6ad348c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.373363 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d2f251602454a1eadf4e2803e226531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.375907 746 tablet_service.cc:1468] Processing CreateTablet for tablet fed43adfae594e3b91ed1012838c3317 (DEFAULT_TABLE table=default.Table_8df215ea93ca49c391a3c3526be6fc22 [id=210e808ee8ce412888a041fa36c91cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.376297 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed43adfae594e3b91ed1012838c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.381954 749 tablet_service.cc:1468] Processing CreateTablet for tablet 07837660a7db47299b8e11f3335cb5d7 (DEFAULT_TABLE table=default.Table_3666ef79a51b4d33a802675181e67fac [id=41cb67813536436fba87f4c8a020e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.382205 750 tablet_service.cc:1468] Processing CreateTablet for tablet a5a5ca65ab3e4f759b0b972aa793732e (DEFAULT_TABLE table=default.Table_e1caebaa6b58419199c7477103213ec9 [id=f99197b69ce0473da6969318c8218094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.382413 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07837660a7db47299b8e11f3335cb5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.381958 748 tablet_service.cc:1468] Processing CreateTablet for tablet 20750adfe4c14c55bddc2ea3bbf99ef2 (DEFAULT_TABLE table=default.Table_a3b92394c95f4ff5bb95019ea9de1300 [id=3a7146572e604811af6284d71852ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.383335 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20750adfe4c14c55bddc2ea3bbf99ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.383195 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 3d2f251602454a1eadf4e2803e226531 (DEFAULT_TABLE table=default.Table_825f0fc01f0a47eaa92ad842fe4e7c68 [id=ac1f155584c44b8491d7997f6ad348c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.385572 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a0342575a2b34bf4b2390968e6acbe5a (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.388191 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 51d1ab1dc05c4758acba0f3f436a3e6f (DEFAULT_TABLE table=default.Table_c35896915c304b1b8dada33cb186ca0e [id=42ab65ca881947ccac05139c8a5e9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.389042 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a5ca65ab3e4f759b0b972aa793732e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.390193 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0f9955688a574c80a8b14c2018ed41f7 (DEFAULT_TABLE table=default.Table_e342d843577548959d8be6a80119e769 [id=fd849b1f759a49f6aa52e840c309cbb8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.393296 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1a5a40faa418420ea31bc1d46cab1f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5a40faa418420ea31bc1d46cab1f4a already in progress: creating tablet
I20250625 07:52:22.394232 754 tablet_service.cc:1468] Processing CreateTablet for tablet 328b21f669c442fb86259500e794d222 (DEFAULT_TABLE table=default.Table_a8813cc630ad48ca87a8bcc192010644 [id=76f35e31204549ddba67624bd1813977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.394666 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b21f669c442fb86259500e794d222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.401058 758 tablet_service.cc:1468] Processing CreateTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.401485 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73561bac3bf4ece9da5433ab4d19b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.407786 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5de0c5bf60b24808af124c95833d9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.408465 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328b21f669c442fb86259500e794d222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.408565 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba (DEFAULT_TABLE table=default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32 [id=a3bf600f08114191807e33fa9f466cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.408932 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3b6dc279984d7fba83a6308fbc0fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.412832 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f903943e9a4151b6fa1e71d1c05473. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.417416 758 tablet_service.cc:1468] Processing CreateTablet for tablet aba6420802914efe85111282398e6270 (DEFAULT_TABLE table=default.Table_40088cb2328f4d629da1e5d5341d135d [id=c836336c95794b4d8e4a4829545b3528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.417917 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba6420802914efe85111282398e6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.420675 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c847e0f7d9cc4199913e73ef2f4302fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.420953 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.421661 755 tablet_service.cc:1468] Processing CreateTablet for tablet d43d98c0c51b40c6bdb0c2705465038e (DEFAULT_TABLE table=default.Table_6a808c6d46a94091b18c21073a499b8a [id=b6489a827b56434380514e9565a4aacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.422122 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43d98c0c51b40c6bdb0c2705465038e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.424849 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8f25febb41c403ca8be268206157756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.424862 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.425314 750 tablet_service.cc:1468] Processing CreateTablet for tablet 9b83f4c8544946d9adb3346df200b215 (DEFAULT_TABLE table=default.Table_73f9bd402c6044089fc883b657511521 [id=9ae1c8331e2149b3a32cf69a11f834c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.425698 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b83f4c8544946d9adb3346df200b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.432865 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (table default.Table_5946e2f2580849578b2bda787459faef [id=e441385dc1044575b17f64b06f14c648]) successfully deleted
I20250625 07:52:22.433102 744 tablet_service.cc:1468] Processing CreateTablet for tablet f1983df4902f48d2b408a99b440368e7 (DEFAULT_TABLE table=default.Table_5d201449036c4c1a8351c46d645de96c [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.435437 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:22.435707 914 raft_consensus.cc:2466] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:22.436610 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1983df4902f48d2b408a99b440368e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.438526 749 tablet_service.cc:1468] Processing CreateTablet for tablet e8750a4e01f449f3abcb0a283da1d9db (DEFAULT_TABLE table=default.Table_66583e344f734087b51f1ea2962f4123 [id=fd09e9ddd8e9468ba7df7083942e1b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.439016 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8750a4e01f449f3abcb0a283da1d9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.456892 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed43adfae594e3b91ed1012838c3317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.464879 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cdde72919a4a3ca29434d475c2d44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.469492 1651 raft_consensus.cc:491] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.469861 1651 raft_consensus.cc:513] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.471675 1651 leader_election.cc:290] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.471918 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.472190 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:22.472908 851 leader_election.cc:343] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:22.473248 850 leader_election.cc:343] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.473418 850 leader_election.cc:304] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.477464 850 connection.cc:537] client connection to 127.0.104.62:38477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:22.477455 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.62:38477 (error 108) [suppressed 5 similar messages]
I20250625 07:52:22.477874 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f9955688a574c80a8b14c2018ed41f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.481324 985 connection.cc:537] client connection to 127.0.104.62:38477 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:52:22.482301 1048 raft_consensus.cc:3058] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:22.483335 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.484483 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d1ab1dc05c4758acba0f3f436a3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.488106 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d2f251602454a1eadf4e2803e226531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.489123 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d65a9a2a6139477ab0b18e19a77822fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.489748 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d65a9a2a6139477ab0b18e19a77822fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:22.490969 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80691f033eb44c6ba256b47acd868b70 consensus state has no leader
I20250625 07:52:22.491309 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_09cf1afe39e740689d3cef2c20f65f84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:22.491588 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0342575a2b34bf4b2390968e6acbe5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.491811 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09cf1afe39e740689d3cef2c20f65f84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.493194 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1fb8a396533b4c55ab45e2dbd46237d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.493700 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fb8a396533b4c55ab45e2dbd46237d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.494999 1651 raft_consensus.cc:2747] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.497817 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_69353bbfa60e4fc594bf8e56bd3f2b17" } new_table_name: "default.Table_37564180b46a46b7bc77bf339f37cafb" modify_external_catalogs: true
W20250625 07:52:22.498340 1163 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.083s sys 0.167s
W20250625 07:52:22.498488 1163 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.610s user 0.083s sys 0.167s
I20250625 07:52:22.499321 1163 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:22.499527 1163 ts_tablet_manager.cc:1397] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.616s user 0.083s sys 0.168s
I20250625 07:52:22.500237 1163 raft_consensus.cc:357] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.500439 1163 raft_consensus.cc:383] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.500507 1163 raft_consensus.cc:738] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.500749 1163 consensus_queue.cc:260] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.501188 1163 ts_tablet_manager.cc:1428] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:22.501473 1163 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:22.502542 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99421eff752142348c8a899e7ac961b1 consensus state has no leader
I20250625 07:52:22.502905 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5a7fc7d8b43c4a7b805952d5fe0e01f7" } new_table_name: "default.Table_d5657e03390941c28681972134ee0159" modify_external_catalogs: true
I20250625 07:52:22.503655 1163 tablet_bootstrap.cc:654] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.507246 1160 ts_tablet_manager.cc:1918] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.507527 1160 log.cc:1199] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/651c74bf9a9b47d6ab5a875a5c304783
I20250625 07:52:22.508055 1160 ts_tablet_manager.cc:1939] T 651c74bf9a9b47d6ab5a875a5c304783 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.510289 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 (DEFAULT_TABLE table=default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5 [id=6f669706429b46738a022387976f26e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.510900 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efffbfca58d44ffa97e56b5c10e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.513469 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f1983df4902f48d2b408a99b440368e7 (DEFAULT_TABLE table=default.Table_5d201449036c4c1a8351c46d645de96c [id=8340c9a7e5874bf486d4f07192ebb908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.514070 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1983df4902f48d2b408a99b440368e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.517217 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a5a5ca65ab3e4f759b0b972aa793732e (DEFAULT_TABLE table=default.Table_e1caebaa6b58419199c7477103213ec9 [id=f99197b69ce0473da6969318c8218094]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.517822 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5a5ca65ab3e4f759b0b972aa793732e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.522835 1163 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:22.523008 1163 ts_tablet_manager.cc:1397] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20250625 07:52:22.523802 1163 raft_consensus.cc:357] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.523991 1163 raft_consensus.cc:383] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.524101 1163 raft_consensus.cc:738] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.524329 1163 consensus_queue.cc:260] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.525072 1163 ts_tablet_manager.cc:1428] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 07:52:22.526017 1163 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:22.527454 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 20750adfe4c14c55bddc2ea3bbf99ef2 (DEFAULT_TABLE table=default.Table_a3b92394c95f4ff5bb95019ea9de1300 [id=3a7146572e604811af6284d71852ed02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.527974 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20750adfe4c14c55bddc2ea3bbf99ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.528385 1163 tablet_bootstrap.cc:654] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.528999 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 07837660a7db47299b8e11f3335cb5d7 (DEFAULT_TABLE table=default.Table_3666ef79a51b4d33a802675181e67fac [id=41cb67813536436fba87f4c8a020e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.529680 1019 tablet_service.cc:1468] Processing CreateTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.530359 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba (DEFAULT_TABLE table=default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32 [id=a3bf600f08114191807e33fa9f466cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.532411 1013 tablet_service.cc:1468] Processing CreateTablet for tablet aba6420802914efe85111282398e6270 (DEFAULT_TABLE table=default.Table_40088cb2328f4d629da1e5d5341d135d [id=c836336c95794b4d8e4a4829545b3528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.537362 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5d019a5abdbe452d98f3d30ce45417c0 (DEFAULT_TABLE table=default.Table_0e621427f14e48a497fcd6d20346cee7 [id=00d1ab8388b24ed4bf735cac083a51e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.538096 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d019a5abdbe452d98f3d30ce45417c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.543298 1163 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:22.543550 1163 ts_tablet_manager.cc:1397] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20250625 07:52:22.544664 1163 raft_consensus.cc:357] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.544968 1163 raft_consensus.cc:383] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.545073 1163 raft_consensus.cc:738] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.545532 1166 ts_tablet_manager.cc:1918] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.545444 1163 consensus_queue.cc:260] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.545799 1166 log.cc:1199] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/49697a10cdac4c8ebb69f10f4c3437ec
W20250625 07:52:22.545809 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea (table default.Table_c250beeb533e41af826edcc10557bf9a [id=86f69fa9857d43729a0f3ad1fc43b61d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d1fc7cb613b6472ea34a0ea7bc8a08ea consensus state has no leader
I20250625 07:52:22.546131 1163 ts_tablet_manager.cc:1428] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:22.546511 1166 ts_tablet_manager.cc:1939] T 49697a10cdac4c8ebb69f10f4c3437ec P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:22.546550 1163 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:22.548447 1163 tablet_bootstrap.cc:654] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:22.549672 889 consensus_meta.cc:287] T 3f1f025775d04da09eda17af4ac4fe10 P 429026e8d98d46339db143f84897a092: Time spent flushing consensus metadata: real 0.610s user 0.006s sys 0.000s
W20250625 07:52:22.556825 1158 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.003s sys 0.012s
W20250625 07:52:22.557031 1158 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.607s user 0.003s sys 0.012s
I20250625 07:52:22.562557 1543 raft_consensus.cc:491] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.562740 1543 raft_consensus.cc:513] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.563452 1543 leader_election.cc:290] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.563627 1543 raft_consensus.cc:491] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.563776 1543 raft_consensus.cc:513] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.564388 1543 leader_election.cc:290] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.567906 1162 ts_tablet_manager.cc:1428] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.274s user 0.001s sys 0.001s
I20250625 07:52:22.568250 1162 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:22.570128 1162 tablet_bootstrap.cc:654] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.577104 1617 raft_consensus.cc:491] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.577270 1617 raft_consensus.cc:513] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.577872 1617 leader_election.cc:290] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:22.578683 713 leader_election.cc:343] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.579015 713 leader_election.cc:304] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.576915 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29eaa8d93e5b4d1d8ed147b60c7da859" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.579319 1617 raft_consensus.cc:2802] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:22.578784 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6382963863548398bed90f03b840806" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.579413 1617 raft_consensus.cc:491] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:22.579512 1617 raft_consensus.cc:3058] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:22.580467 1158 tablet_bootstrap.cc:492] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:22.580823 1158 ts_tablet_manager.cc:1397] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.848s user 0.005s sys 0.014s
I20250625 07:52:22.581688 1158 raft_consensus.cc:357] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:22.581925 1158 raft_consensus.cc:383] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.582067 1158 raft_consensus.cc:738] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.582209 1617 raft_consensus.cc:513] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.582515 1158 consensus_queue.cc:260] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:22.582787 1617 leader_election.cc:290] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:22.583801 713 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 29eaa8d93e5b4d1d8ed147b60c7da859
W20250625 07:52:22.583986 713 leader_election.cc:343] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b6382963863548398bed90f03b840806
I20250625 07:52:22.585676 1158 ts_tablet_manager.cc:1428] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:22.585994 1158 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:22.588364 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07837660a7db47299b8e11f3335cb5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.589429 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f73561bac3bf4ece9da5433ab4d19b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.590224 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3b6dc279984d7fba83a6308fbc0fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.590780 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba6420802914efe85111282398e6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.616355 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet dfc22a46594f47deb8b142bc14c81a0b
I20250625 07:52:22.618619 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fc290bab1df44405ab85e0058a8bc848"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.619383 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc290bab1df44405ab85e0058a8bc848 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.620919 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet aba6420802914efe85111282398e6270
I20250625 07:52:22.621605 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8b3050d2440e4b81907a06a8574cf631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.622181 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b3050d2440e4b81907a06a8574cf631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.623903 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 962633b0d9644c308f0b7fe1db714daf
I20250625 07:52:22.624630 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a0c38fff8ede45cd8909ea2002e45c37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.625149 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0c38fff8ede45cd8909ea2002e45c37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.626914 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 506794253c84486faa926de885c48c58
I20250625 07:52:22.627539 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e5cc8a2851a74b148a50729e9402a687"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.628036 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5cc8a2851a74b148a50729e9402a687 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.630352 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5ebb69eae9414c4ea940ee210526f79a
I20250625 07:52:22.630965 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d057d5343c334e9e92d5d9d977f90e6a" } new_table_name: "default.Table_bb53a82b1c824335b04789e50a88c91f" modify_external_catalogs: true
I20250625 07:52:22.640339 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e8750a4e01f449f3abcb0a283da1d9db (DEFAULT_TABLE table=default.Table_66583e344f734087b51f1ea2962f4123 [id=fd09e9ddd8e9468ba7df7083942e1b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.649247 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 571) took 1037 ms. Trace:
I20250625 07:52:22.653496 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 563) took 1252 ms. Trace:
I20250625 07:52:22.662709 1160 rpcz_store.cc:276] 0625 07:52:21.611980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.612039 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:22.112671 (+500632us) ts_tablet_manager.cc:1145] Deleting tablet 651c74bf9a9b47d6ab5a875a5c304783
0625 07:52:22.112679 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.223999 (+111320us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.507084 (+283085us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.649145 (+142061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":500355}
I20250625 07:52:22.669689 1166 rpcz_store.cc:276] 0625 07:52:21.400526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.400588 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:22.189294 (+788706us) ts_tablet_manager.cc:1145] Deleting tablet 49697a10cdac4c8ebb69f10f4c3437ec
0625 07:52:22.189303 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.289366 (+100063us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.545416 (+256050us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.653394 (+107978us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105766,"tablet-delete.queue_time_us":788447}
I20250625 07:52:22.672626 1160 tablet_replica.cc:331] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.681579 1160 raft_consensus.cc:2241] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.682025 1160 raft_consensus.cc:2270] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.686247 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 651c74bf9a9b47d6ab5a875a5c304783 (table default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]) successfully deleted
W20250625 07:52:22.688427 583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477): Couldn't send request to peer 00a3faeffaa4422ea73b98ee74768a4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:22.683627 889 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc22a46594f47deb8b142bc14c81a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.684507 893 tablet_service.cc:1515] Processing DeleteTablet for tablet aba6420802914efe85111282398e6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.684895 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 962633b0d9644c308f0b7fe1db714daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.685344 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 506794253c84486faa926de885c48c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.685738 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ebb69eae9414c4ea940ee210526f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.711298 1160 ts_tablet_manager.cc:1905] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.715409 1651 raft_consensus.cc:491] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.715766 1651 raft_consensus.cc:513] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.717701 1651 leader_election.cc:290] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:22.717324 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10c7f4b4187c428bbec136c85219da4d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:22.721033 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 571) took 1107 ms. Trace:
I20250625 07:52:22.721197 1166 rpcz_store.cc:276] 0625 07:52:21.613358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.613554 (+ 196us) service_pool.cc:225] Handling call
0625 07:52:22.720819 (+1107265us) ts_tablet_manager.cc:1145] Deleting tablet 17be34eba92246df973f3d464e13fd30
0625 07:52:22.720829 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.720972 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107027}
W20250625 07:52:22.721316 851 leader_election.cc:343] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.721655 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 572) took 1108 ms. Trace:
I20250625 07:52:22.721766 1166 rpcz_store.cc:276] 0625 07:52:21.613507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.613990 (+ 483us) service_pool.cc:225] Handling call
0625 07:52:22.721482 (+1107492us) ts_tablet_manager.cc:1145] Deleting tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e
0625 07:52:22.721492 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.721580 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107324}
I20250625 07:52:22.722117 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 573) took 1099 ms. Trace:
I20250625 07:52:22.722249 1166 rpcz_store.cc:276] 0625 07:52:21.622275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.622462 (+ 187us) service_pool.cc:225] Handling call
0625 07:52:22.721978 (+1099516us) ts_tablet_manager.cc:1145] Deleting tablet 35a93e9198d1417bb1b8e00560921483
0625 07:52:22.721986 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.722057 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099294}
I20250625 07:52:22.722645 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 574) took 1100 ms. Trace:
I20250625 07:52:22.722759 1166 rpcz_store.cc:276] 0625 07:52:21.622412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.622736 (+ 324us) service_pool.cc:225] Handling call
0625 07:52:22.722508 (+1099772us) ts_tablet_manager.cc:1145] Deleting tablet 3758b4fc14df40f3bb9a6e77699981ea
0625 07:52:22.722515 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.722587 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099617}
I20250625 07:52:22.723089 1166 tablet_replica.cc:331] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:22.723284 1166 raft_consensus.cc:2241] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.723481 1166 raft_consensus.cc:2270] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.724048 1166 ts_tablet_manager.cc:1905] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.724525 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 5de1492e483e47919ee5581ecee227be (table default.Table_0ef24d5b4b124360b654d6903b994151 [id=23322aa9321140c39272b5b2a102fb02]). Expected version 2 got 0
I20250625 07:52:22.725083 593 master_service.cc:948] ReplaceTablet: received request to replace tablet fc85ec9546eb4da99b877c603d6b91e6 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:22.725708 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_38f6c0a3b6e644eeb000b074c6393e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:22.726269 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38f6c0a3b6e644eeb000b074c6393e2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.727422 1162 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:22.641151 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d43d98c0c51b40c6bdb0c2705465038e (DEFAULT_TABLE table=default.Table_6a808c6d46a94091b18c21073a499b8a [id=b6489a827b56434380514e9565a4aacc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.727587 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65231ee1a59d49eb96ba352b74c3460c consensus state has no leader
I20250625 07:52:22.641673 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.727948 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet aba6420802914efe85111282398e6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.728179 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 962633b0d9644c308f0b7fe1db714daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.728364 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 506794253c84486faa926de885c48c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.728533 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ebb69eae9414c4ea940ee210526f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.642230 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.642936 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 9b83f4c8544946d9adb3346df200b215 (DEFAULT_TABLE table=default.Table_73f9bd402c6044089fc883b657511521 [id=9ae1c8331e2149b3a32cf69a11f834c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.646051 1012 consensus_meta.cc:287] T 64a5ff005b424ae5ae9831263abf58ef P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent flushing consensus metadata: real 0.503s user 0.008s sys 0.001s
I20250625 07:52:22.707046 1158 tablet_bootstrap.cc:654] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:22.729058 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b4627f4da6b4542aff1e44157087b14 consensus state has no leader
I20250625 07:52:22.729411 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fe4cc985dba64833998e95782e5634a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:22.718623 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10c7f4b4187c428bbec136c85219da4d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:22.729938 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe4cc985dba64833998e95782e5634a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:22.730327 850 leader_election.cc:343] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.730500 850 leader_election.cc:304] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.730998 1651 raft_consensus.cc:2747] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:22.731791 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.720402 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet c6031b9695cd4a99bd0e5391663170ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.705600 1666 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:22.737005 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a564b726340d44f485906f969425fdd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:22.737818 1163 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:22.738005 1163 ts_tablet_manager.cc:1397] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.192s user 0.060s sys 0.080s
W20250625 07:52:22.738000 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a564b726340d44f485906f969425fdd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:22.738292 1162 ts_tablet_manager.cc:1397] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.000s
I20250625 07:52:22.739037 1163 raft_consensus.cc:357] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.739282 1163 raft_consensus.cc:383] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.739404 1163 raft_consensus.cc:738] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.739691 1163 consensus_queue.cc:260] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:22.740422 1163 ts_tablet_manager.cc:1428] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:22.741740 1163 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:22.743351 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 49697a10cdac4c8ebb69f10f4c3437ec (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:22.743611 1163 tablet_bootstrap.cc:654] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:22.744092 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 17be34eba92246df973f3d464e13fd30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17be34eba92246df973f3d464e13fd30 already in progress: creating tablet
I20250625 07:52:22.744421 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
W20250625 07:52:22.745030 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e already in progress: creating tablet
W20250625 07:52:22.745702 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 35a93e9198d1417bb1b8e00560921483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a93e9198d1417bb1b8e00560921483 already in progress: creating tablet
W20250625 07:52:22.746179 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3758b4fc14df40f3bb9a6e77699981ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3758b4fc14df40f3bb9a6e77699981ea already in progress: creating tablet
I20250625 07:52:22.746628 1162 raft_consensus.cc:357] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.748507 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc22a46594f47deb8b142bc14c81a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:22.746728 742 tablet_service.cc:1515] Processing DeleteTablet for tablet dfc22a46594f47deb8b142bc14c81a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.747141 756 tablet_service.cc:1515] Processing DeleteTablet for tablet aba6420802914efe85111282398e6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.747664 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 506794253c84486faa926de885c48c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.747804 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 962633b0d9644c308f0b7fe1db714daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.748893 1162 raft_consensus.cc:383] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.748989 1162 raft_consensus.cc:738] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.749118 915 raft_consensus.cc:3058] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:22.749032 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d9d66546d7c42949dfe5db77f1057fe" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:22.750998 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b3f2915b351407da08a9d6ba8ed402f" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:22.751222 1160 ts_tablet_manager.cc:1918] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.751360 1160 log.cc:1199] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b5f917cfd2384a72920847f8b89358b9
I20250625 07:52:22.751751 1160 ts_tablet_manager.cc:1939] T b5f917cfd2384a72920847f8b89358b9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.752200 915 raft_consensus.cc:2466] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:22.752555 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 570) took 1140 ms. Trace:
W20250625 07:52:22.748118 713 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.752663 1160 rpcz_store.cc:276] 0625 07:52:21.611795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.614267 (+ 2472us) service_pool.cc:225] Handling call
0625 07:52:22.672463 (+1058196us) ts_tablet_manager.cc:1145] Deleting tablet b5f917cfd2384a72920847f8b89358b9
0625 07:52:22.672483 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.715541 (+ 43058us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.751053 (+ 35512us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.752519 (+ 1466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1057829}
I20250625 07:52:22.752846 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ebb69eae9414c4ea940ee210526f79a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.752930 713 leader_election.cc:304] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:22.753098 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b5f917cfd2384a72920847f8b89358b9 (table default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]) successfully deleted
I20250625 07:52:22.753221 1617 raft_consensus.cc:2747] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:22.753439 713 leader_election.cc:343] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.753593 713 leader_election.cc:304] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.753779 713 raft_consensus.cc:2684] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:22.754829 713 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:22.756211 713 leader_election.cc:304] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:22.756472 1660 raft_consensus.cc:2802] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:22.756626 1660 raft_consensus.cc:695] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:22.756960 1660 consensus_queue.cc:237] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.757746 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 569) took 1146 ms. Trace:
I20250625 07:52:22.757869 1160 rpcz_store.cc:276] 0625 07:52:21.611675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.614642 (+ 2967us) service_pool.cc:225] Handling call
0625 07:52:22.757560 (+1142918us) ts_tablet_manager.cc:1145] Deleting tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e
0625 07:52:22.757573 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.757704 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142614}
I20250625 07:52:22.758265 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 568) took 1146 ms. Trace:
I20250625 07:52:22.758358 1160 rpcz_store.cc:276] 0625 07:52:21.611565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.616045 (+ 4480us) service_pool.cc:225] Handling call
0625 07:52:22.758138 (+1142093us) ts_tablet_manager.cc:1145] Deleting tablet 17be34eba92246df973f3d464e13fd30
0625 07:52:22.758147 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.758237 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135705}
I20250625 07:52:22.758689 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 573) took 1135 ms. Trace:
I20250625 07:52:22.758790 1160 rpcz_store.cc:276] 0625 07:52:21.623341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.625219 (+ 1878us) service_pool.cc:225] Handling call
0625 07:52:22.758585 (+1133366us) ts_tablet_manager.cc:1145] Deleting tablet 3758b4fc14df40f3bb9a6e77699981ea
0625 07:52:22.758595 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.758661 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133090}
I20250625 07:52:22.762739 1162 consensus_queue.cc:260] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.763298 1162 ts_tablet_manager.cc:1428] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250625 07:52:22.763636 1162 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:22.764740 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd0eb2e6d9e45eeafd0aa2d109fce6e already in progress: creating tablet
W20250625 07:52:22.765053 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 17be34eba92246df973f3d464e13fd30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17be34eba92246df973f3d464e13fd30 already in progress: creating tablet
W20250625 07:52:22.765252 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3758b4fc14df40f3bb9a6e77699981ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3758b4fc14df40f3bb9a6e77699981ea already in progress: creating tablet
I20250625 07:52:22.765383 1162 tablet_bootstrap.cc:654] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.765388 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 572) took 1135 ms. Trace:
I20250625 07:52:22.765718 1160 rpcz_store.cc:276] 0625 07:52:21.623218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.625537 (+ 2319us) service_pool.cc:225] Handling call
0625 07:52:22.758963 (+1133426us) ts_tablet_manager.cc:1145] Deleting tablet 35a93e9198d1417bb1b8e00560921483
0625 07:52:22.758972 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.759066 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133250}
W20250625 07:52:22.766192 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 35a93e9198d1417bb1b8e00560921483 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35a93e9198d1417bb1b8e00560921483 already in progress: creating tablet
I20250625 07:52:22.766458 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 582) took 1050 ms. Trace:
I20250625 07:52:22.766568 1160 rpcz_store.cc:276] 0625 07:52:21.716404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.716796 (+ 392us) service_pool.cc:225] Handling call
0625 07:52:22.765985 (+1049189us) ts_tablet_manager.cc:1145] Deleting tablet 5b04a575e0984f879f0179c8baa3f3fa
0625 07:52:22.765996 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.766429 (+ 433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048961}
W20250625 07:52:22.767006 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5b04a575e0984f879f0179c8baa3f3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b04a575e0984f879f0179c8baa3f3fa already in progress: creating tablet
I20250625 07:52:22.768674 1160 tablet_replica.cc:331] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.768937 1160 raft_consensus.cc:2241] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.769105 1160 raft_consensus.cc:2270] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.769665 1160 ts_tablet_manager.cc:1905] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.769889 1158 tablet_bootstrap.cc:492] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:22.770110 1158 ts_tablet_manager.cc:1397] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.184s user 0.006s sys 0.002s
I20250625 07:52:22.770996 1158 raft_consensus.cc:357] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:22.771263 1158 raft_consensus.cc:383] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.771385 1158 raft_consensus.cc:738] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.771724 1158 consensus_queue.cc:260] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:22.772550 1158 ts_tablet_manager.cc:1428] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:22.773007 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c91554bb23646638781392298543d03" } new_table_name: "default.Table_c2ec45c7324942d39b6da745f3ca5e1f" modify_external_catalogs: true
I20250625 07:52:22.778277 881 tablet_service.cc:1468] Processing CreateTablet for tablet ef471c579fc142c8a111b91165d97c5c (DEFAULT_TABLE table=default.Table_e925686f85164a9b94c4c280ebebfadb [id=aa9123512ed546648335a8e901727953]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.778296 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a3b92394c95f4ff5bb95019ea9de1300" } new_table_name: "default.Table_6ddaef2306f14d4ba08e26ee4d1822b6" modify_external_catalogs: true
I20250625 07:52:22.778769 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef471c579fc142c8a111b91165d97c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.779722 883 tablet_service.cc:1468] Processing CreateTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 (DEFAULT_TABLE table=default.Table_94af0d30f3b8498c8bd98dbeec6e5085 [id=686c2711221c4e769795e258573164f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.780121 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81451d3745f24cbd9a56307c557e5ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.781015 886 tablet_service.cc:1468] Processing CreateTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 (DEFAULT_TABLE table=default.Table_53313429425a455cb97296e7d4b6f0dd [id=9990ab905fce42b996788a673dd1de64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.781361 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1764c3d48a7546f7828ee93aa51dcb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.784996 885 tablet_service.cc:1468] Processing CreateTablet for tablet 71c66d9107024afa8b68deeac22332f6 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.785364 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34544157495f4e14a7978e0eec70b522 consensus state has no leader
I20250625 07:52:22.785416 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c66d9107024afa8b68deeac22332f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.785730 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 0929ed76622248d1a41f08f4258108cd from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:22.786226 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 35a93e9198d1417bb1b8e00560921483 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:22.788985 889 tablet_service.cc:1468] Processing CreateTablet for tablet 953cfe2c224043eca4da735c2a5dbebc (DEFAULT_TABLE table=default.Table_a85de53a06a54f77a07282be4323dd74 [id=48d892390cd947f6a4296f10b76770b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.789371 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953cfe2c224043eca4da735c2a5dbebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.784999 891 tablet_service.cc:1468] Processing CreateTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 (DEFAULT_TABLE table=default.Table_ed4947df987b4ce48a2209dea3bf6dee [id=336314e4b3794fbbb135aa303e415df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.793654 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 381ae5ef5df9437d9bfe0a51987dfe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.796578 1166 ts_tablet_manager.cc:1918] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.796730 1166 log.cc:1199] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b6382963863548398bed90f03b840806
I20250625 07:52:22.788597 893 tablet_service.cc:1468] Processing CreateTablet for tablet 7832aebe55544771b518a4f0800333a8 (DEFAULT_TABLE table=default.Table_1f83aa0aa3e8436281b7c9886cdc096c [id=c06f9eb245c54efbba4e8fc294a01a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.797112 1166 ts_tablet_manager.cc:1939] T b6382963863548398bed90f03b840806 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:22.797195 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7832aebe55544771b518a4f0800333a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.797871 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 584) took 1072 ms. Trace:
I20250625 07:52:22.802233 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.797991 1166 rpcz_store.cc:276] 0625 07:52:21.725056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.725115 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:22.722990 (+997875us) ts_tablet_manager.cc:1145] Deleting tablet b6382963863548398bed90f03b840806
0625 07:52:22.723001 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.743799 (+ 20798us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.796426 (+ 52627us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.797834 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997616}
I20250625 07:52:22.806035 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 583) took 1088 ms. Trace:
I20250625 07:52:22.806674 1166 rpcz_store.cc:276] 0625 07:52:21.717860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.725164 (+ 7304us) service_pool.cc:225] Handling call
0625 07:52:22.805867 (+1080703us) ts_tablet_manager.cc:1145] Deleting tablet 5b04a575e0984f879f0179c8baa3f3fa
0625 07:52:22.805881 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.805999 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080459}
I20250625 07:52:22.810313 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 586) took 1058 ms. Trace:
I20250625 07:52:22.812726 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fc85ec9546eb4da99b877c603d6b91e6
I20250625 07:52:22.813015 1166 rpcz_store.cc:276] 0625 07:52:21.752063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.756776 (+ 4713us) service_pool.cc:225] Handling call
0625 07:52:22.809241 (+1052465us) ts_tablet_manager.cc:1145] Deleting tablet be7b755d95b14ecea7a691003455f381
0625 07:52:22.809252 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.810276 (+ 1024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":994646}
I20250625 07:52:22.813246 593 catalog_manager.cc:6652] ReplaceTablet: tablet fc85ec9546eb4da99b877c603d6b91e6 deleted and replaced by tablet b4d308b0b16a4ab6a3a3589087c94136
I20250625 07:52:22.813663 892 tablet_service.cc:1515] Processing DeleteTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 with delete_type TABLET_DATA_DELETED (replaced by tablet b4d308b0b16a4ab6a3a3589087c94136) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:22.814451 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896419bd407f43e1a0a361330ff9821b consensus state has no leader
I20250625 07:52:22.819832 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 587) took 1067 ms. Trace:
W20250625 07:52:22.820710 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20750adfe4c14c55bddc2ea3bbf99ef2 consensus state has no leader
I20250625 07:52:22.824941 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8c340466a42e47dead226f74c8176f49" } modify_external_catalogs: true
I20250625 07:52:22.824903 1166 rpcz_store.cc:276] 0625 07:52:21.752273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.752376 (+ 103us) service_pool.cc:225] Handling call
0625 07:52:22.819648 (+1067272us) ts_tablet_manager.cc:1145] Deleting tablet e06ecad6c7464ffe8ec6f632634307c9
0625 07:52:22.819661 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.819792 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993073}
I20250625 07:52:22.825148 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8c340466a42e47dead226f74c8176f49" } modify_external_catalogs: true
I20250625 07:52:22.825402 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 585) took 1073 ms. Trace:
I20250625 07:52:22.825505 1166 rpcz_store.cc:276] 0625 07:52:21.751928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.752146 (+ 218us) service_pool.cc:225] Handling call
0625 07:52:22.825210 (+1073064us) ts_tablet_manager.cc:1145] Deleting tablet 9f31104720614b028e373ae1500c31cc
0625 07:52:22.825222 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.825349 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998586}
I20250625 07:52:22.825963 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c62e57a0d535414a93d82dca7dd477ba" } modify_external_catalogs: true
I20250625 07:52:22.826023 1166 tablet_replica.cc:331] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:22.826216 1166 raft_consensus.cc:2241] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.826190 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c62e57a0d535414a93d82dca7dd477ba" } modify_external_catalogs: true
I20250625 07:52:22.826406 1166 raft_consensus.cc:2270] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.826990 1166 ts_tablet_manager.cc:1905] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.830577 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.830996 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_525806c470a840e8b54ce05f09aaf215" } modify_external_catalogs: true
I20250625 07:52:22.831210 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_525806c470a840e8b54ce05f09aaf215" } modify_external_catalogs: true
I20250625 07:52:22.831822 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c" } modify_external_catalogs: true
I20250625 07:52:22.832034 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c" } modify_external_catalogs: true
I20250625 07:52:22.833214 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f4184bdee984426ea3612ca34ca1bf92" } modify_external_catalogs: true
I20250625 07:52:22.833398 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f4184bdee984426ea3612ca34ca1bf92" } modify_external_catalogs: true
W20250625 07:52:22.834410 1670 log.cc:927] Time spent T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250625 07:52:22.836839 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 (DEFAULT_TABLE table=default.Table_53313429425a455cb97296e7d4b6f0dd [id=9990ab905fce42b996788a673dd1de64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.837462 594 master_service.cc:948] ReplaceTablet: received request to replace tablet b5bc347c278c44c69444caf9ded72e1a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:22.837550 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 71c66d9107024afa8b68deeac22332f6 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.838099 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 7832aebe55544771b518a4f0800333a8 (DEFAULT_TABLE table=default.Table_1f83aa0aa3e8436281b7c9886cdc096c [id=c06f9eb245c54efbba4e8fc294a01a20]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:22.841794 713 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.842042 713 leader_election.cc:304] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:22.842574 713 raft_consensus.cc:2684] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:22.845273 713 leader_election.cc:343] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:22.846177 1163 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:22.846364 1163 ts_tablet_manager.cc:1397] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.106s user 0.007s sys 0.003s
I20250625 07:52:22.847908 1163 raft_consensus.cc:357] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.848142 1163 raft_consensus.cc:383] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.848245 1163 raft_consensus.cc:738] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.848549 1163 consensus_queue.cc:260] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.849061 1163 ts_tablet_manager.cc:1428] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:22.849331 1163 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:22.850982 1163 tablet_bootstrap.cc:654] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.838110 1022 tablet_service.cc:1468] Processing CreateTablet for tablet ef471c579fc142c8a111b91165d97c5c (DEFAULT_TABLE table=default.Table_e925686f85164a9b94c4c280ebebfadb [id=aa9123512ed546648335a8e901727953]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.852671 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b6382963863548398bed90f03b840806 (table default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]) successfully deleted
W20250625 07:52:22.853008 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5b04a575e0984f879f0179c8baa3f3fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b04a575e0984f879f0179c8baa3f3fa already in progress: creating tablet
W20250625 07:52:22.853289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet be7b755d95b14ecea7a691003455f381 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be7b755d95b14ecea7a691003455f381 already in progress: creating tablet
W20250625 07:52:22.853502 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e06ecad6c7464ffe8ec6f632634307c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e06ecad6c7464ffe8ec6f632634307c9 already in progress: creating tablet
W20250625 07:52:22.853737 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9f31104720614b028e373ae1500c31cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f31104720614b028e373ae1500c31cc already in progress: creating tablet
W20250625 07:52:22.853933 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d76681f162074b67a6b886887b5a3d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76681f162074b67a6b886887b5a3d61 already in progress: creating tablet
W20250625 07:52:22.854184 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ccddd630a82341cba7a857e67a349fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccddd630a82341cba7a857e67a349fe0 already in progress: creating tablet
I20250625 07:52:22.857244 1160 ts_tablet_manager.cc:1918] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.857409 1160 log.cc:1199] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/21658216da3544a2b8b8f39667782ffb
I20250625 07:52:22.857764 1160 ts_tablet_manager.cc:1939] T 21658216da3544a2b8b8f39667782ffb P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.858632 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 583) took 1136 ms. Trace:
I20250625 07:52:22.858721 1160 rpcz_store.cc:276] 0625 07:52:21.722594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.722650 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:22.766792 (+1044142us) ts_tablet_manager.cc:1145] Deleting tablet 21658216da3544a2b8b8f39667782ffb
0625 07:52:22.766800 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.838362 (+ 71562us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.857093 (+ 18731us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.858614 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043783}
I20250625 07:52:22.859113 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 585) took 1129 ms. Trace:
I20250625 07:52:22.859197 1160 rpcz_store.cc:276] 0625 07:52:21.729899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.729955 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:22.858980 (+1129025us) ts_tablet_manager.cc:1145] Deleting tablet 9f31104720614b028e373ae1500c31cc
0625 07:52:22.858990 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.859094 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128801}
I20250625 07:52:22.859591 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 586) took 1128 ms. Trace:
I20250625 07:52:22.859669 1160 rpcz_store.cc:276] 0625 07:52:21.731074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.732745 (+ 1671us) service_pool.cc:225] Handling call
0625 07:52:22.859480 (+1126735us) ts_tablet_manager.cc:1145] Deleting tablet e06ecad6c7464ffe8ec6f632634307c9
0625 07:52:22.859488 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.859577 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126449}
I20250625 07:52:22.859912 1160 tablet_replica.cc:331] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.860091 1160 raft_consensus.cc:2241] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.860229 1160 raft_consensus.cc:2270] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.860497 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7832aebe55544771b518a4f0800333a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.861444 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0102a3b176e146a8b19abe22d3aba656
I20250625 07:52:22.861544 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0929ed76622248d1a41f08f4258108cd
I20250625 07:52:22.862089 590 catalog_manager.cc:6652] ReplaceTablet: tablet 0929ed76622248d1a41f08f4258108cd deleted and replaced by tablet 5099f01a4a124d23a7c32accbd576e54
I20250625 07:52:22.862488 883 tablet_service.cc:1468] Processing CreateTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 (DEFAULT_TABLE table=default.Table_1fb8a396533b4c55ab45e2dbd46237d7 [id=82fb6680169b4956b0b62c1e71b20f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.862874 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48bc90f80f14bc2b35ce85acafa9028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.863140 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 21658216da3544a2b8b8f39667782ffb (table default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]) successfully deleted
W20250625 07:52:22.863405 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9f31104720614b028e373ae1500c31cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f31104720614b028e373ae1500c31cc already in progress: creating tablet
W20250625 07:52:22.863624 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e06ecad6c7464ffe8ec6f632634307c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e06ecad6c7464ffe8ec6f632634307c9 already in progress: creating tablet
I20250625 07:52:22.867717 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef471c579fc142c8a111b91165d97c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.867887 749 tablet_service.cc:1468] Processing CreateTablet for tablet ef471c579fc142c8a111b91165d97c5c (DEFAULT_TABLE table=default.Table_e925686f85164a9b94c4c280ebebfadb [id=aa9123512ed546648335a8e901727953]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.868793 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef471c579fc142c8a111b91165d97c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.869107 743 tablet_service.cc:1468] Processing CreateTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 (DEFAULT_TABLE table=default.Table_53313429425a455cb97296e7d4b6f0dd [id=9990ab905fce42b996788a673dd1de64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.869537 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1764c3d48a7546f7828ee93aa51dcb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.877394 753 tablet_service.cc:1468] Processing CreateTablet for tablet 71c66d9107024afa8b68deeac22332f6 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.877859 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c66d9107024afa8b68deeac22332f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.878883 1166 ts_tablet_manager.cc:1918] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.879038 1166 log.cc:1199] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:22.879120 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.879364 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.879493 1166 ts_tablet_manager.cc:1939] T 7d9d66546d7c42949dfe5db77f1057fe P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:22.879526 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0102a3b176e146a8b19abe22d3aba656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:22.879882 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b (DEFAULT_TABLE table=default.Table_fc290bab1df44405ab85e0058a8bc848 [id=78db5f7a86bd45ff837949c9dba5250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.880255 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.880534 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 594) took 1027 ms. Trace:
I20250625 07:52:22.881888 1160 ts_tablet_manager.cc:1905] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.883677 891 tablet_service.cc:1468] Processing CreateTablet for tablet 77b8f4e99b9c4c4c838c97d89b361f96 (DEFAULT_TABLE table=default.Table_d65a9a2a6139477ab0b18e19a77822fa [id=005ab54379974835b4a44ca2d1c1d8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.884059 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8f4e99b9c4c4c838c97d89b361f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.885046 753 tablet_service.cc:1468] Processing CreateTablet for tablet 7832aebe55544771b518a4f0800333a8 (DEFAULT_TABLE table=default.Table_1f83aa0aa3e8436281b7c9886cdc096c [id=c06f9eb245c54efbba4e8fc294a01a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.885514 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7832aebe55544771b518a4f0800333a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.888538 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6a6fab215b63465a9e48738d3f27651e (DEFAULT_TABLE table=default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5 [id=dfe2b8ce653b4649bf8b45ea76d4cdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.889065 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a6fab215b63465a9e48738d3f27651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.890069 742 tablet_service.cc:1468] Processing CreateTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 (DEFAULT_TABLE table=default.Table_ed4947df987b4ce48a2209dea3bf6dee [id=336314e4b3794fbbb135aa303e415df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.890105 749 tablet_service.cc:1468] Processing CreateTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 (DEFAULT_TABLE table=default.Table_94af0d30f3b8498c8bd98dbeec6e5085 [id=686c2711221c4e769795e258573164f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.890633 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 381ae5ef5df9437d9bfe0a51987dfe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.890911 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81451d3745f24cbd9a56307c557e5ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.892977 881 tablet_service.cc:1468] Processing CreateTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 (DEFAULT_TABLE table=default.Table_e5cc8a2851a74b148a50729e9402a687 [id=d920e0540b62419fa62c2de64a015a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.893437 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cded78f0ce7a4af696a4dcaf7b4f5051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.897696 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.898185 756 tablet_service.cc:1515] Processing DeleteTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 with delete_type TABLET_DATA_DELETED (replaced by tablet b4d308b0b16a4ab6a3a3589087c94136) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.898932 756 tablet_service.cc:1468] Processing CreateTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 (DEFAULT_TABLE table=default.Table_e5cc8a2851a74b148a50729e9402a687 [id=d920e0540b62419fa62c2de64a015a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.899308 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cded78f0ce7a4af696a4dcaf7b4f5051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.899782 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8750a4e01f449f3abcb0a283da1d9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.880654 1166 rpcz_store.cc:276] 0625 07:52:21.852939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.853060 (+ 121us) service_pool.cc:225] Handling call
0625 07:52:22.825971 (+972911us) ts_tablet_manager.cc:1145] Deleting tablet 7d9d66546d7c42949dfe5db77f1057fe
0625 07:52:22.825976 (+ 5us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.861223 (+ 35247us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.878765 (+ 17542us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.880453 (+ 1688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971787}
I20250625 07:52:22.905859 1161 tablet_replica.cc:331] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:22.906103 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 0102a3b176e146a8b19abe22d3aba656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.906277 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43d98c0c51b40c6bdb0c2705465038e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.906412 1161 raft_consensus.cc:2241] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.906380 746 tablet_service.cc:1468] Processing CreateTablet for tablet 77b8f4e99b9c4c4c838c97d89b361f96 (DEFAULT_TABLE table=default.Table_d65a9a2a6139477ab0b18e19a77822fa [id=005ab54379974835b4a44ca2d1c1d8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.906601 1161 raft_consensus.cc:2270] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.906605 754 tablet_service.cc:1468] Processing CreateTablet for tablet 6a6fab215b63465a9e48738d3f27651e (DEFAULT_TABLE table=default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5 [id=dfe2b8ce653b4649bf8b45ea76d4cdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.907805 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b83f4c8544946d9adb3346df200b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.908893 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 595) took 1030 ms. Trace:
I20250625 07:52:22.909008 1166 rpcz_store.cc:276] 0625 07:52:21.873445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.873507 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:22.901322 (+1027815us) ts_tablet_manager.cc:1145] Deleting tablet 486463a8140c4e769cc79870d2acff78
0625 07:52:22.901665 (+ 343us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.903428 (+ 1763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027225}
I20250625 07:52:22.909176 1161 ts_tablet_manager.cc:1905] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.910354 1166 tablet_replica.cc:331] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:22.910564 1166 raft_consensus.cc:2241] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.910776 1166 raft_consensus.cc:2270] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.911338 1166 ts_tablet_manager.cc:1905] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.911540 755 tablet_service.cc:1468] Processing CreateTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 (DEFAULT_TABLE table=default.Table_1fb8a396533b4c55ab45e2dbd46237d7 [id=82fb6680169b4956b0b62c1e71b20f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.911592 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:22.911836 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8f4e99b9c4c4c838c97d89b361f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.912068 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48bc90f80f14bc2b35ce85acafa9028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.912345 758 tablet_service.cc:1468] Processing CreateTablet for tablet 33466d4481c24294b5cc7a7736a1d737 (DEFAULT_TABLE table=default.Table_abc25ebdf79c4455be53d3dfb090fdd6 [id=f2578edceb6742e78bf4126e62aee4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.912760 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33466d4481c24294b5cc7a7736a1d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.911876 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b (DEFAULT_TABLE table=default.Table_fc290bab1df44405ab85e0058a8bc848 [id=78db5f7a86bd45ff837949c9dba5250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.917088 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a6fab215b63465a9e48738d3f27651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.917394 893 tablet_service.cc:1468] Processing CreateTablet for tablet 33466d4481c24294b5cc7a7736a1d737 (DEFAULT_TABLE table=default.Table_abc25ebdf79c4455be53d3dfb090fdd6 [id=f2578edceb6742e78bf4126e62aee4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.917783 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33466d4481c24294b5cc7a7736a1d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.919693 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.923913 743 tablet_service.cc:1468] Processing CreateTablet for tablet 953cfe2c224043eca4da735c2a5dbebc (DEFAULT_TABLE table=default.Table_a85de53a06a54f77a07282be4323dd74 [id=48d892390cd947f6a4296f10b76770b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.924444 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953cfe2c224043eca4da735c2a5dbebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.927696 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7a798b2471104d69b814df95edc541c6 (DEFAULT_TABLE table=default.Table_09cf1afe39e740689d3cef2c20f65f84 [id=482fb8b5f46c4fa496f062c9ad007a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.928577 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a798b2471104d69b814df95edc541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.928990 1162 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:22.929188 1162 ts_tablet_manager.cc:1397] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.000s
I20250625 07:52:22.930140 1162 raft_consensus.cc:357] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.933773 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1764c3d48a7546f7828ee93aa51dcb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.935424 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c66d9107024afa8b68deeac22332f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.936468 883 tablet_service.cc:1468] Processing CreateTablet for tablet 3522ef3ed0714b60944a851bad7375fc (DEFAULT_TABLE table=default.Table_06a0b466c0dc496badbf8a1b18535c72 [id=340eb1704b6048fc81145f0b7b9caa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.938108 1160 ts_tablet_manager.cc:1918] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.938393 1160 log.cc:1199] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/31942f0998bb4118a91f476325851d5c
I20250625 07:52:22.938789 1160 ts_tablet_manager.cc:1939] T 31942f0998bb4118a91f476325851d5c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.939620 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 591) took 1146 ms. Trace:
I20250625 07:52:22.939770 1160 rpcz_store.cc:276] 0625 07:52:21.793006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.793309 (+ 303us) service_pool.cc:225] Handling call
0625 07:52:22.859872 (+1066563us) ts_tablet_manager.cc:1145] Deleting tablet 31942f0998bb4118a91f476325851d5c
0625 07:52:22.859880 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.922568 (+ 62688us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.937849 (+ 15281us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.939596 (+ 1747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1066182}
I20250625 07:52:22.940183 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7d9d66546d7c42949dfe5db77f1057fe (table default.Table_3b26428e16b14c4aaaf95f0f92b3670c [id=60fd5cf37619424381543d3cdc94167d]) successfully deleted
I20250625 07:52:22.940294 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 31942f0998bb4118a91f476325851d5c (table default.Table_a28f5f4927f34e03b2ea4784b65cfe43 [id=7686e223b6fb47e79ecb2257550a64b5]) successfully deleted
I20250625 07:52:22.940590 1162 raft_consensus.cc:383] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.940884 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 592) took 1137 ms. Trace:
I20250625 07:52:22.941069 1160 rpcz_store.cc:276] 0625 07:52:21.803192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.803402 (+ 210us) service_pool.cc:225] Handling call
0625 07:52:22.940742 (+1137340us) ts_tablet_manager.cc:1145] Deleting tablet d76681f162074b67a6b886887b5a3d61
0625 07:52:22.940753 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.940867 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136966}
W20250625 07:52:22.941545 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d76681f162074b67a6b886887b5a3d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d76681f162074b67a6b886887b5a3d61 already in progress: creating tablet
W20250625 07:52:22.941751 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 486463a8140c4e769cc79870d2acff78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 486463a8140c4e769cc79870d2acff78 already in progress: creating tablet
I20250625 07:52:22.941906 1158 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:22.942122 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 584) took 1213 ms. Trace:
I20250625 07:52:22.941432 1162 raft_consensus.cc:738] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.942241 1160 rpcz_store.cc:276] 0625 07:52:21.728264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.826355 (+ 98091us) service_pool.cc:225] Handling call
0625 07:52:22.942003 (+1115648us) ts_tablet_manager.cc:1145] Deleting tablet be7b755d95b14ecea7a691003455f381
0625 07:52:22.942012 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.942108 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101662}
W20250625 07:52:22.942607 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1a5a40faa418420ea31bc1d46cab1f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5a40faa418420ea31bc1d46cab1f4a already in progress: creating tablet
W20250625 07:52:22.942726 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet be7b755d95b14ecea7a691003455f381 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be7b755d95b14ecea7a691003455f381 already in progress: creating tablet
I20250625 07:52:22.943288 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 593) took 1117 ms. Trace:
I20250625 07:52:22.943406 1160 rpcz_store.cc:276] 0625 07:52:21.826072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.826128 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:22.943170 (+1117042us) ts_tablet_manager.cc:1145] Deleting tablet ccddd630a82341cba7a857e67a349fe0
0625 07:52:22.943180 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.943272 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075400}
W20250625 07:52:22.943645 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44bfcc2b20ba4d0b83641988f2c0ee35 already in progress: creating tablet
I20250625 07:52:22.943876 1158 tablet_bootstrap.cc:654] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:22.943933 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ccddd630a82341cba7a857e67a349fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccddd630a82341cba7a857e67a349fe0 already in progress: creating tablet
I20250625 07:52:22.944411 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 594) took 1070 ms. Trace:
I20250625 07:52:22.944775 1160 rpcz_store.cc:276] 0625 07:52:21.874439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.874498 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:22.944298 (+1069800us) ts_tablet_manager.cc:1145] Deleting tablet 486463a8140c4e769cc79870d2acff78
0625 07:52:22.944309 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.944396 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069489}
W20250625 07:52:22.944924 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d393a85f5fde4b3fb57e5b3ec372791c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d393a85f5fde4b3fb57e5b3ec372791c already in progress: creating tablet
W20250625 07:52:22.945408 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 486463a8140c4e769cc79870d2acff78 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 486463a8140c4e769cc79870d2acff78 already in progress: creating tablet
I20250625 07:52:22.938097 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3522ef3ed0714b60944a851bad7375fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:22.946255 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet aba6420802914efe85111282398e6270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba6420802914efe85111282398e6270 already in progress: creating tablet
I20250625 07:52:22.946588 1160 tablet_replica.cc:331] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.946982 1160 raft_consensus.cc:2241] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:22.947256 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet dfc22a46594f47deb8b142bc14c81a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc22a46594f47deb8b142bc14c81a0b already in progress: creating tablet
I20250625 07:52:22.947503 1160 raft_consensus.cc:2270] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:22.948340 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 506794253c84486faa926de885c48c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506794253c84486faa926de885c48c58 already in progress: creating tablet
I20250625 07:52:22.949071 1160 ts_tablet_manager.cc:1905] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:22.954949 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 962633b0d9644c308f0b7fe1db714daf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962633b0d9644c308f0b7fe1db714daf already in progress: creating tablet
I20250625 07:52:22.946583 1162 consensus_queue.cc:260] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:22.961369 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5ebb69eae9414c4ea940ee210526f79a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebb69eae9414c4ea940ee210526f79a already in progress: creating tablet
I20250625 07:52:22.962302 1162 ts_tablet_manager.cc:1428] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20250625 07:52:22.965144 758 tablet_service.cc:1468] Processing CreateTablet for tablet 7a798b2471104d69b814df95edc541c6 (DEFAULT_TABLE table=default.Table_09cf1afe39e740689d3cef2c20f65f84 [id=482fb8b5f46c4fa496f062c9ad007a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.965291 1162 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:22.965834 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a798b2471104d69b814df95edc541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.967154 1162 tablet_bootstrap.cc:654] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.967358 1160 ts_tablet_manager.cc:1918] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.967463 1160 log.cc:1199] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/445bfed532004108a12c8fd78370c1cb
I20250625 07:52:22.967765 1160 ts_tablet_manager.cc:1939] T 445bfed532004108a12c8fd78370c1cb P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.970535 742 tablet_service.cc:1468] Processing CreateTablet for tablet 3522ef3ed0714b60944a851bad7375fc (DEFAULT_TABLE table=default.Table_06a0b466c0dc496badbf8a1b18535c72 [id=340eb1704b6048fc81145f0b7b9caa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:22.970626 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 601) took 1077 ms. Trace:
I20250625 07:52:22.970734 1160 rpcz_store.cc:276] 0625 07:52:21.893005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.907157 (+ 14152us) service_pool.cc:225] Handling call
0625 07:52:22.946520 (+1039363us) ts_tablet_manager.cc:1145] Deleting tablet 445bfed532004108a12c8fd78370c1cb
0625 07:52:22.946530 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.952744 (+ 6214us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.967252 (+ 14508us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:22.970600 (+ 3348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039152}
I20250625 07:52:22.971421 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3522ef3ed0714b60944a851bad7375fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:22.971889 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 445bfed532004108a12c8fd78370c1cb (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
I20250625 07:52:22.972390 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 602) took 1059 ms. Trace:
I20250625 07:52:22.972724 1485 raft_consensus.cc:491] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:22.972978 1160 rpcz_store.cc:276] 0625 07:52:21.913305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:21.913356 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:22.972255 (+1058899us) ts_tablet_manager.cc:1145] Deleting tablet 1a5a40faa418420ea31bc1d46cab1f4a
0625 07:52:22.972267 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:22.972369 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058657}
I20250625 07:52:22.973403 1485 raft_consensus.cc:513] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
W20250625 07:52:22.973640 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1a5a40faa418420ea31bc1d46cab1f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5a40faa418420ea31bc1d46cab1f4a already in progress: creating tablet
I20250625 07:52:22.974081 1485 leader_election.cc:290] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:22.974411 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d393a85f5fde4b3fb57e5b3ec372791c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d393a85f5fde4b3fb57e5b3ec372791c already in progress: creating tablet
W20250625 07:52:22.974938 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44bfcc2b20ba4d0b83641988f2c0ee35 already in progress: creating tablet
W20250625 07:52:22.975392 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet dfc22a46594f47deb8b142bc14c81a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc22a46594f47deb8b142bc14c81a0b already in progress: creating tablet
W20250625 07:52:22.975647 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet aba6420802914efe85111282398e6270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aba6420802914efe85111282398e6270 already in progress: creating tablet
W20250625 07:52:22.975826 986 leader_election.cc:343] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 53c6c51a90754878977aeeaacf4773a6
W20250625 07:52:22.975984 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 962633b0d9644c308f0b7fe1db714daf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962633b0d9644c308f0b7fe1db714daf already in progress: creating tablet
I20250625 07:52:22.976429 1160 tablet_replica.cc:331] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:22.976595 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 506794253c84486faa926de885c48c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506794253c84486faa926de885c48c58 already in progress: creating tablet
W20250625 07:52:22.976941 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5ebb69eae9414c4ea940ee210526f79a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebb69eae9414c4ea940ee210526f79a already in progress: creating tablet
I20250625 07:52:22.977033 1160 raft_consensus.cc:2241] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:22.977175 1160 raft_consensus.cc:2270] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:22.978199 1166 ts_tablet_manager.cc:1918] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.978354 1166 log.cc:1199] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:22.978758 1166 ts_tablet_manager.cc:1939] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:22.978808 1162 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:22.978960 1162 ts_tablet_manager.cc:1397] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250625 07:52:22.979809 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (table default.Table_97384ebbc1584955af0a6bc425084522 [id=131a7d8cbeb34a0a8db53be305d31ccd]) successfully deleted
I20250625 07:52:22.979777 1162 raft_consensus.cc:357] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.980001 1162 raft_consensus.cc:383] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:22.980101 1162 raft_consensus.cc:738] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:22.981472 1160 ts_tablet_manager.cc:1905] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:22.983481 1162 consensus_queue.cc:260] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:22.985390 1162 ts_tablet_manager.cc:1428] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
W20250625 07:52:22.985713 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fc85ec9546eb4da99b877c603d6b91e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc85ec9546eb4da99b877c603d6b91e6 already in progress: creating tablet
I20250625 07:52:22.986080 1162 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:22.987299 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0102a3b176e146a8b19abe22d3aba656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0102a3b176e146a8b19abe22d3aba656 already in progress: creating tablet
W20250625 07:52:22.988178 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0929ed76622248d1a41f08f4258108cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0929ed76622248d1a41f08f4258108cd already in progress: creating tablet
I20250625 07:52:22.993643 1162 tablet_bootstrap.cc:654] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:22.996815 1160 ts_tablet_manager.cc:1918] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:22.997155 1160 log.cc:1199] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:22.997557 1160 ts_tablet_manager.cc:1939] T 10c7f4b4187c428bbec136c85219da4d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:22.998755 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 10c7f4b4187c428bbec136c85219da4d (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:22.999612 1160 tablet_replica.cc:331] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:22.999958 1160 raft_consensus.cc:2241] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.000418 1160 raft_consensus.cc:2270] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.002444 1160 ts_tablet_manager.cc:1905] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.002841 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 (DEFAULT_TABLE table=default.Table_94af0d30f3b8498c8bd98dbeec6e5085 [id=686c2711221c4e769795e258573164f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.003480 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81451d3745f24cbd9a56307c557e5ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.001638 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 953cfe2c224043eca4da735c2a5dbebc (DEFAULT_TABLE table=default.Table_a85de53a06a54f77a07282be4323dd74 [id=48d892390cd947f6a4296f10b76770b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.008834 1162 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.009285 1162 ts_tablet_manager.cc:1397] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20250625 07:52:23.010435 1162 raft_consensus.cc:357] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.011277 1162 raft_consensus.cc:383] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.011447 1162 raft_consensus.cc:738] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.011821 1162 consensus_queue.cc:260] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.013015 1162 ts_tablet_manager.cc:1428] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:23.013839 1162 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.015668 1162 tablet_bootstrap.cc:654] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.023286 1651 raft_consensus.cc:491] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.023494 1651 raft_consensus.cc:513] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:23.024109 1160 ts_tablet_manager.cc:1918] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.024307 1160 log.cc:1199] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/802c22a6346442c88abbeb4b9ef20c57
I20250625 07:52:23.024821 1160 ts_tablet_manager.cc:1939] T 802c22a6346442c88abbeb4b9ef20c57 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:23.025013 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0518fbc7f52454296ae79a3c92ef800" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:23.025837 851 leader_election.cc:343] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.024345 1651 leader_election.cc:290] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.029901 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 802c22a6346442c88abbeb4b9ef20c57 (table default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]) successfully deleted
W20250625 07:52:23.031042 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fc85ec9546eb4da99b877c603d6b91e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc85ec9546eb4da99b877c603d6b91e6 already in progress: creating tablet
I20250625 07:52:23.032090 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8441cf3142424cc78e9fd8690dd8752e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.033579 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fe9a633f1a8e450ba2a6db9adc198734"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.034085 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe9a633f1a8e450ba2a6db9adc198734 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.035326 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9650a59cdcb04bf395e5cc36090a99f1" } new_table_name: "default.Table_3a7a4153cc2f4ba0a19445f60f6703ca" modify_external_catalogs: true
I20250625 07:52:23.037727 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2a4d9d86ac944371a48bcdd835f6c025" } new_table_name: "default.Table_44af03fefc224c18ac4dc64986b4fa3d" modify_external_catalogs: true
I20250625 07:52:23.038465 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a87d4f6703a94cbb96962177f2bb0cd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.038962 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a87d4f6703a94cbb96962177f2bb0cd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:23.032665 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8441cf3142424cc78e9fd8690dd8752e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.046169 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b5bc347c278c44c69444caf9ded72e1a
I20250625 07:52:23.046753 594 catalog_manager.cc:6652] ReplaceTablet: tablet b5bc347c278c44c69444caf9ded72e1a deleted and replaced by tablet 9b66fad15471497390fc753b4015bb8f
I20250625 07:52:23.047439 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bc347c278c44c69444caf9ded72e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 9b66fad15471497390fc753b4015bb8f) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.047906 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bc347c278c44c69444caf9ded72e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 9b66fad15471497390fc753b4015bb8f) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:23.048792 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b5bc347c278c44c69444caf9ded72e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bc347c278c44c69444caf9ded72e1a already in progress: creating tablet
I20250625 07:52:23.050976 1161 ts_tablet_manager.cc:1918] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.052156 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet fc85ec9546eb4da99b877c603d6b91e6 with delete_type TABLET_DATA_DELETED (replaced by tablet b4d308b0b16a4ab6a3a3589087c94136) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.052431 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 953cfe2c224043eca4da735c2a5dbebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.053478 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c3d8b6b5c06248879e6052f6ae7fe46e
I20250625 07:52:23.054338 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f35dcb866b4e4dcc98712f798468a51e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.054971 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f35dcb866b4e4dcc98712f798468a51e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.056238 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e10149f4ae0645ba88508ca51ae9f77a
I20250625 07:52:23.057013 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b6fbe0d023e745a0a9cc130f361967d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.057051 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 (DEFAULT_TABLE table=default.Table_ed4947df987b4ce48a2209dea3bf6dee [id=336314e4b3794fbbb135aa303e415df1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.057552 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6fbe0d023e745a0a9cc130f361967d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.057687 742 tablet_service.cc:1515] Processing DeleteTablet for tablet e10149f4ae0645ba88508ca51ae9f77a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.057744 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0518fbc7f52454296ae79a3c92ef800" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:23.058415 850 leader_election.cc:343] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:23.058563 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e10149f4ae0645ba88508ca51ae9f77a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10149f4ae0645ba88508ca51ae9f77a already in progress: creating tablet
I20250625 07:52:23.058532 850 leader_election.cc:304] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.058779 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet dddbc1200acf4e7fa93ef85b398d5832
I20250625 07:52:23.059005 1651 raft_consensus.cc:2747] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.059794 1160 tablet_replica.cc:331] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:23.059978 1160 raft_consensus.cc:2241] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.060122 1160 raft_consensus.cc:2270] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.060488 885 tablet_service.cc:1515] Processing DeleteTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.060993 742 tablet_service.cc:1515] Processing DeleteTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.061749 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7480c07af7ce4a3889bd8d42d7419d57
W20250625 07:52:23.062251 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f655669b74b4a6594fbd63c45d15bf4 consensus state has no leader
I20250625 07:52:23.062335 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c3c1fb488db2488da0ad1bf71b323567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.062448 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_602fb37488c14e66b4cfc551eb323c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.062726 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b3786a3e21ca46cdb5365fe9bc606a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.062916 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3c1fb488db2488da0ad1bf71b323567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:23.063174 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3786a3e21ca46cdb5365fe9bc606a7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.063627 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_817356d9efcf4f889897f9e942387859" } new_table_name: "default.Table_634beccfc1a34257bd0bc04d1d020530" modify_external_catalogs: true
W20250625 07:52:23.065153 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet dddbc1200acf4e7fa93ef85b398d5832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dddbc1200acf4e7fa93ef85b398d5832 already in progress: creating tablet
I20250625 07:52:23.065486 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:23.062917 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_602fb37488c14e66b4cfc551eb323c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:23.066181 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7480c07af7ce4a3889bd8d42d7419d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7480c07af7ce4a3889bd8d42d7419d57 already in progress: creating tablet
I20250625 07:52:23.068054 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.068230 891 tablet_service.cc:1515] Processing DeleteTablet for tablet e10149f4ae0645ba88508ca51ae9f77a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.069110 881 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.069514 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:23.070350 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c3d8b6b5c06248879e6052f6ae7fe46e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d8b6b5c06248879e6052f6ae7fe46e already in progress: creating tablet
I20250625 07:52:23.071313 1013 tablet_service.cc:1468] Processing CreateTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 (DEFAULT_TABLE table=default.Table_e5cc8a2851a74b148a50729e9402a687 [id=d920e0540b62419fa62c2de64a015a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.071316 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b (DEFAULT_TABLE table=default.Table_fc290bab1df44405ab85e0058a8bc848 [id=78db5f7a86bd45ff837949c9dba5250f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.073886 1015 tablet_service.cc:1468] Processing CreateTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 (DEFAULT_TABLE table=default.Table_1fb8a396533b4c55ab45e2dbd46237d7 [id=82fb6680169b4956b0b62c1e71b20f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.075657 1543 raft_consensus.cc:491] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.075838 1543 raft_consensus.cc:513] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.076076 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 77b8f4e99b9c4c4c838c97d89b361f96 (DEFAULT_TABLE table=default.Table_d65a9a2a6139477ab0b18e19a77822fa [id=005ab54379974835b4a44ca2d1c1d8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.076110 1161 log.cc:1199] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c6031b9695cd4a99bd0e5391663170ff
I20250625 07:52:23.076555 1543 leader_election.cc:290] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:23.077499 713 leader_election.cc:343] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5c28a064006145eea000e74da10d0794
I20250625 07:52:23.078287 1161 ts_tablet_manager.cc:1939] T c6031b9695cd4a99bd0e5391663170ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:23.079262 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0102a3b176e146a8b19abe22d3aba656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.079526 1162 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.079783 1162 ts_tablet_manager.cc:1397] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.001s
I20250625 07:52:23.080312 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.080705 1162 raft_consensus.cc:357] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.080979 1162 raft_consensus.cc:383] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.080964 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 381ae5ef5df9437d9bfe0a51987dfe62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.081120 1162 raft_consensus.cc:738] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.081319 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cded78f0ce7a4af696a4dcaf7b4f5051. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.081534 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b8f4e99b9c4c4c838c97d89b361f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.081475 1162 consensus_queue.cc:260] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.081916 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.081966 1162 ts_tablet_manager.cc:1428] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:23.084813 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b48bc90f80f14bc2b35ce85acafa9028. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.091634 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3522ef3ed0714b60944a851bad7375fc (DEFAULT_TABLE table=default.Table_06a0b466c0dc496badbf8a1b18535c72 [id=340eb1704b6048fc81145f0b7b9caa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.092159 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3522ef3ed0714b60944a851bad7375fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.092731 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 33466d4481c24294b5cc7a7736a1d737 (DEFAULT_TABLE table=default.Table_abc25ebdf79c4455be53d3dfb090fdd6 [id=f2578edceb6742e78bf4126e62aee4b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.093057 1162 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.094942 1162 tablet_bootstrap.cc:654] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.096896 1160 ts_tablet_manager.cc:1905] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.098145 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6d71f14d5bd44891a4c2e98f3f43034e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.098784 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d71f14d5bd44891a4c2e98f3f43034e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.098884 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cdfa29007d814e609da9a03c52f558f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.099341 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdfa29007d814e609da9a03c52f558f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.100421 1163 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:23.098294 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7f82e83ed924d2bbedd33d96642a10a consensus state has no leader
I20250625 07:52:23.100612 1163 ts_tablet_manager.cc:1397] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.251s user 0.026s sys 0.028s
I20250625 07:52:23.100973 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f562abe21a314a6aaf1bf1574b9b908d" } new_table_name: "default.Table_ce3de32cf03042d6990e81e54f461669" modify_external_catalogs: true
I20250625 07:52:23.101332 1163 raft_consensus.cc:357] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.101487 1163 raft_consensus.cc:383] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.101547 1163 raft_consensus.cc:738] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.102006 1163 consensus_queue.cc:260] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.105634 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6a6fab215b63465a9e48738d3f27651e (DEFAULT_TABLE table=default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5 [id=dfe2b8ce653b4649bf8b45ea76d4cdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.106922 1162 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.107151 1162 ts_tablet_manager.cc:1397] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250625 07:52:23.107333 1163 ts_tablet_manager.cc:1428] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 07:52:23.107999 1162 raft_consensus.cc:357] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.108261 1162 raft_consensus.cc:383] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.108419 1162 raft_consensus.cc:738] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.108747 1162 consensus_queue.cc:260] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.109477 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7a798b2471104d69b814df95edc541c6 (DEFAULT_TABLE table=default.Table_09cf1afe39e740689d3cef2c20f65f84 [id=482fb8b5f46c4fa496f062c9ad007a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.110461 1162 ts_tablet_manager.cc:1428] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:23.111711 1162 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.112094 1163 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:23.113983 1163 tablet_bootstrap.cc:654] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.114034 1162 tablet_bootstrap.cc:654] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.115440 1160 ts_tablet_manager.cc:1918] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.115557 1160 log.cc:1199] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a0518fbc7f52454296ae79a3c92ef800
I20250625 07:52:23.115753 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33466d4481c24294b5cc7a7736a1d737. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.115868 1160 ts_tablet_manager.cc:1939] T a0518fbc7f52454296ae79a3c92ef800 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:23.116812 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bc347c278c44c69444caf9ded72e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 9b66fad15471497390fc753b4015bb8f) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.117060 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a0518fbc7f52454296ae79a3c92ef800 (table default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]) successfully deleted
I20250625 07:52:23.117383 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a6fab215b63465a9e48738d3f27651e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.119043 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a798b2471104d69b814df95edc541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.119558 1162 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.119834 1162 ts_tablet_manager.cc:1397] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
W20250625 07:52:23.120064 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0929ed76622248d1a41f08f4258108cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0929ed76622248d1a41f08f4258108cd already in progress: creating tablet
I20250625 07:52:23.120575 1160 tablet_replica.cc:331] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:23.120676 1162 raft_consensus.cc:357] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.120831 1160 raft_consensus.cc:2241] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.120971 1162 raft_consensus.cc:383] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.121004 1160 raft_consensus.cc:2270] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.121068 1162 raft_consensus.cc:738] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.121323 1162 consensus_queue.cc:260] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.122104 1160 ts_tablet_manager.cc:1905] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.123769 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e6509d5448ab4dba96e3a6571d4963a7" } new_table_name: "default.Table_9aed63ffa6e74306b79919180fb2f25f" modify_external_catalogs: true
I20250625 07:52:23.124775 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5" } modify_external_catalogs: true
I20250625 07:52:23.124956 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f170413ab2ac4ec6b713e5fdf1fae5b5" } modify_external_catalogs: true
I20250625 07:52:23.125314 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a5a40faa418420ea31bc1d46cab1f4a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:23.125149 594 catalog_manager.cc:5582] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:23.125586 594 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
W20250625 07:52:23.126470 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f04016cda7f746d49e30be3b1c9c18b6 consensus state has no leader
I20250625 07:52:23.132522 891 tablet_service.cc:1468] Processing CreateTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 (DEFAULT_TABLE table=default.Table_38f6c0a3b6e644eeb000b074c6393e2f [id=b4646bd77e9e4b3ba8fac8d10736993c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.133054 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7dd3b2533244b93a88d12660ea0a4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.133519 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5099f01a4a124d23a7c32accbd576e54 (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.133981 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5099f01a4a124d23a7c32accbd576e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.134281 1162 ts_tablet_manager.cc:1428] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250625 07:52:23.134795 1162 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.136557 1162 tablet_bootstrap.cc:654] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.136937 885 tablet_service.cc:1468] Processing CreateTablet for tablet 5099f01a4a124d23a7c32accbd576e54 (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.137445 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5099f01a4a124d23a7c32accbd576e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.138522 758 tablet_service.cc:1468] Processing CreateTablet for tablet eb98ab2850d746a8bf4733efbcc8bdbc (DEFAULT_TABLE table=default.Table_8b3050d2440e4b81907a06a8574cf631 [id=fada2b1620394efcbeb97fb82a901bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.138729 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 (DEFAULT_TABLE table=default.Table_fe4cc985dba64833998e95782e5634a9 [id=5e69b19f12c144e194afdc5ffc0cccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.138895 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb98ab2850d746a8bf4733efbcc8bdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.139106 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.139384 746 tablet_service.cc:1468] Processing CreateTablet for tablet c2bcee2c83054f538c1f3ea05140685f (DEFAULT_TABLE table=default.Table_a564b726340d44f485906f969425fdd5 [id=246be3a9808f4cc39fac5995cb36fd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.139760 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bcee2c83054f538c1f3ea05140685f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.140790 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.146677 1651 raft_consensus.cc:491] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.146834 1651 raft_consensus.cc:513] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.147424 1651 leader_election.cc:290] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.148000 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "250175f54ba94a9fbc2455d9cf27fb2e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:23.148584 850 leader_election.cc:343] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.151453 1163 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:23.151669 1163 ts_tablet_manager.cc:1397] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250625 07:52:23.151722 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 (DEFAULT_TABLE table=default.Table_fe4cc985dba64833998e95782e5634a9 [id=5e69b19f12c144e194afdc5ffc0cccce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.152176 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.152571 1163 raft_consensus.cc:357] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.152782 1163 raft_consensus.cc:383] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.152851 1163 raft_consensus.cc:738] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.152837 883 tablet_service.cc:1468] Processing CreateTablet for tablet daf0c3772e7941a0a684a29ce160a851 (DEFAULT_TABLE table=default.Table_a0c38fff8ede45cd8909ea2002e45c37 [id=ebe087130038414a95760bcc3bf593b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.153154 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf0c3772e7941a0a684a29ce160a851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.153090 1163 consensus_queue.cc:260] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.153587 1163 ts_tablet_manager.cc:1428] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:23.153861 1163 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:23.155395 1163 tablet_bootstrap.cc:654] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.157593 754 tablet_service.cc:1468] Processing CreateTablet for tablet daf0c3772e7941a0a684a29ce160a851 (DEFAULT_TABLE table=default.Table_a0c38fff8ede45cd8909ea2002e45c37 [id=ebe087130038414a95760bcc3bf593b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.157680 886 tablet_service.cc:1468] Processing CreateTablet for tablet c2bcee2c83054f538c1f3ea05140685f (DEFAULT_TABLE table=default.Table_a564b726340d44f485906f969425fdd5 [id=246be3a9808f4cc39fac5995cb36fd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.157753 753 tablet_service.cc:1468] Processing CreateTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.158015 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bcee2c83054f538c1f3ea05140685f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.159080 893 tablet_service.cc:1468] Processing CreateTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.159453 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d308b0b16a4ab6a3a3589087c94136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.160130 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.160996 889 tablet_service.cc:1468] Processing CreateTablet for tablet eb98ab2850d746a8bf4733efbcc8bdbc (DEFAULT_TABLE table=default.Table_8b3050d2440e4b81907a06a8574cf631 [id=fada2b1620394efcbeb97fb82a901bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.161151 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.161581 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb98ab2850d746a8bf4733efbcc8bdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.162542 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet e10149f4ae0645ba88508ca51ae9f77a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.180651 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.181110 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 (DEFAULT_TABLE table=default.Table_38f6c0a3b6e644eeb000b074c6393e2f [id=b4646bd77e9e4b3ba8fac8d10736993c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.182035 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7dd3b2533244b93a88d12660ea0a4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.183777 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.148002 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "250175f54ba94a9fbc2455d9cf27fb2e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:23.189859 1017 tablet_service.cc:1468] Processing CreateTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 (DEFAULT_TABLE table=default.Table_f442ff326d064398b181ffcb29aa6439 [id=c8f7f6a1e10b4a45905049e48f7cd7c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.191355 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 (DEFAULT_TABLE table=default.Table_fe4cc985dba64833998e95782e5634a9 [id=5e69b19f12c144e194afdc5ffc0cccce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.191691 851 leader_election.cc:343] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.191831 851 leader_election.cc:304] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.192294 1651 raft_consensus.cc:2747] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:23.193066 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d393a85f5fde4b3fb57e5b3ec372791c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d393a85f5fde4b3fb57e5b3ec372791c already in progress: creating tablet
W20250625 07:52:23.193434 583 catalog_manager.cc:4895] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet aba6420802914efe85111282398e6270 because the tablet was not found. No further retry: Not found: Tablet not found: aba6420802914efe85111282398e6270
W20250625 07:52:23.193909 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba24bbd0c4694930978a7750f07601d9 consensus state has no leader
W20250625 07:52:23.193905 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 962633b0d9644c308f0b7fe1db714daf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962633b0d9644c308f0b7fe1db714daf already in progress: creating tablet
W20250625 07:52:23.194168 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 506794253c84486faa926de885c48c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506794253c84486faa926de885c48c58 already in progress: creating tablet
W20250625 07:52:23.194393 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5ebb69eae9414c4ea940ee210526f79a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ebb69eae9414c4ea940ee210526f79a already in progress: creating tablet
W20250625 07:52:23.194579 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44bfcc2b20ba4d0b83641988f2c0ee35 already in progress: creating tablet
I20250625 07:52:23.195410 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c6031b9695cd4a99bd0e5391663170ff (table default.Table_d138706726e44916b4369a0f255a420c [id=513d21f718084c7ebe8e0e4b59a1c7f8]) successfully deleted
W20250625 07:52:23.195637 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet dfc22a46594f47deb8b142bc14c81a0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfc22a46594f47deb8b142bc14c81a0b already in progress: creating tablet
W20250625 07:52:23.195827 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fc85ec9546eb4da99b877c603d6b91e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc85ec9546eb4da99b877c603d6b91e6 already in progress: creating tablet
I20250625 07:52:23.195926 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2efffbfca58d44ffa97e56b5c10e9f29
W20250625 07:52:23.196388 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0102a3b176e146a8b19abe22d3aba656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0102a3b176e146a8b19abe22d3aba656 already in progress: creating tablet
W20250625 07:52:23.196775 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0929ed76622248d1a41f08f4258108cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0929ed76622248d1a41f08f4258108cd already in progress: creating tablet
W20250625 07:52:23.197546 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b5bc347c278c44c69444caf9ded72e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bc347c278c44c69444caf9ded72e1a already in progress: creating tablet
W20250625 07:52:23.197857 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c3d8b6b5c06248879e6052f6ae7fe46e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d8b6b5c06248879e6052f6ae7fe46e already in progress: creating tablet
W20250625 07:52:23.198055 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e10149f4ae0645ba88508ca51ae9f77a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10149f4ae0645ba88508ca51ae9f77a already in progress: creating tablet
W20250625 07:52:23.198179 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a29a9ba27ab4eb494e1b91e4b33aa75 consensus state has no leader
W20250625 07:52:23.198458 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7480c07af7ce4a3889bd8d42d7419d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7480c07af7ce4a3889bd8d42d7419d57 already in progress: creating tablet
W20250625 07:52:23.198725 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet dddbc1200acf4e7fa93ef85b398d5832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dddbc1200acf4e7fa93ef85b398d5832 already in progress: creating tablet
I20250625 07:52:23.199019 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 5099f01a4a124d23a7c32accbd576e54 (DEFAULT_TABLE table=default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.199725 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.200804 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf0c3772e7941a0a684a29ce160a851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.201141 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d308b0b16a4ab6a3a3589087c94136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.200829 1162 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.201422 1162 ts_tablet_manager.cc:1397] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.001s
I20250625 07:52:23.201797 746 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.202327 1162 raft_consensus.cc:357] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.202549 1162 raft_consensus.cc:383] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.202693 1162 raft_consensus.cc:738] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.202940 1162 consensus_queue.cc:260] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.203516 1162 ts_tablet_manager.cc:1428] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:23.203825 1162 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.204995 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.205619 1166 tablet_replica.cc:331] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.205854 1166 raft_consensus.cc:2241] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.206056 1166 raft_consensus.cc:2270] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.206687 1166 ts_tablet_manager.cc:1905] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.212891 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d308b0b16a4ab6a3a3589087c94136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.213071 1015 tablet_service.cc:1468] Processing CreateTablet for tablet eb98ab2850d746a8bf4733efbcc8bdbc (DEFAULT_TABLE table=default.Table_8b3050d2440e4b81907a06a8574cf631 [id=fada2b1620394efcbeb97fb82a901bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.213274 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.213889 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.213850 1012 tablet_service.cc:1468] Processing CreateTablet for tablet daf0c3772e7941a0a684a29ce160a851 (DEFAULT_TABLE table=default.Table_a0c38fff8ede45cd8909ea2002e45c37 [id=ebe087130038414a95760bcc3bf593b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.214196 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf0c3772e7941a0a684a29ce160a851. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.220788 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb98ab2850d746a8bf4733efbcc8bdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.220839 1162 tablet_bootstrap.cc:654] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.214996 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c2bcee2c83054f538c1f3ea05140685f (DEFAULT_TABLE table=default.Table_a564b726340d44f485906f969425fdd5 [id=246be3a9808f4cc39fac5995cb36fd64]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.222396 985 leader_election.cc:343] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 53c6c51a90754878977aeeaacf4773a6
I20250625 07:52:23.212750 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c28a064006145eea000e74da10d0794" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.226409 1160 ts_tablet_manager.cc:1918] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.226528 1160 log.cc:1199] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9418b81f010d4f369d3409e9364bc5fa
I20250625 07:52:23.226783 1160 ts_tablet_manager.cc:1939] T 9418b81f010d4f369d3409e9364bc5fa P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:23.230113 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2bcee2c83054f538c1f3ea05140685f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.232750 758 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.235579 1166 ts_tablet_manager.cc:1918] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.235967 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5099f01a4a124d23a7c32accbd576e54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.236310 1166 log.cc:1199] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/de9ccb0a2d3846228acbc2769786a059
I20250625 07:52:23.235826 743 tablet_service.cc:1468] Processing CreateTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 (DEFAULT_TABLE table=default.Table_38f6c0a3b6e644eeb000b074c6393e2f [id=b4646bd77e9e4b3ba8fac8d10736993c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.237074 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7dd3b2533244b93a88d12660ea0a4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.237478 1163 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:23.237644 1163 ts_tablet_manager.cc:1397] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.016s
I20250625 07:52:23.237989 1166 ts_tablet_manager.cc:1939] T de9ccb0a2d3846228acbc2769786a059 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.238571 1163 raft_consensus.cc:357] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.238752 1163 raft_consensus.cc:383] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.238816 1163 raft_consensus.cc:738] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.239053 1163 consensus_queue.cc:260] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.239440 985 leader_election.cc:304] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:23.239626 1163 ts_tablet_manager.cc:1428] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:23.242835 1485 raft_consensus.cc:2747] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.244166 1166 tablet_replica.cc:331] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.244361 1166 raft_consensus.cc:2241] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:23.244437 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 46980.8us
I20250625 07:52:23.245018 1166 raft_consensus.cc:2270] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:23.245199 713 leader_election.cc:343] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.245492 713 leader_election.cc:304] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.245754 1543 raft_consensus.cc:2747] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.246325 1166 ts_tablet_manager.cc:1905] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.249006 1163 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:23.250706 1163 tablet_bootstrap.cc:654] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.252091 1162 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.252250 1162 ts_tablet_manager.cc:1397] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.003s
I20250625 07:52:23.253079 1162 raft_consensus.cc:357] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.253270 1162 raft_consensus.cc:383] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.253358 1162 raft_consensus.cc:738] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.253638 1162 consensus_queue.cc:260] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.254386 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.255127 758 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.256071 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.256639 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.257123 758 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.258482 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet de9ccb0a2d3846228acbc2769786a059 (table default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]) successfully deleted
W20250625 07:52:23.258744 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2efffbfca58d44ffa97e56b5c10e9f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2efffbfca58d44ffa97e56b5c10e9f29 already in progress: creating tablet
I20250625 07:52:23.260381 1162 ts_tablet_manager.cc:1428] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250625 07:52:23.260738 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9418b81f010d4f369d3409e9364bc5fa (table default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]) successfully deleted
I20250625 07:52:23.261240 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--webserver_interface=127.0.104.62
--webserver_port=39255
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250625 07:52:23.262028 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0102a3b176e146a8b19abe22d3aba656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0102a3b176e146a8b19abe22d3aba656 already in progress: creating tablet
I20250625 07:52:23.262372 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_16d107cb53e442eaa68e834ce2509d3f" } modify_external_catalogs: true
I20250625 07:52:23.262588 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_16d107cb53e442eaa68e834ce2509d3f" } modify_external_catalogs: true
W20250625 07:52:23.262625 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b5bc347c278c44c69444caf9ded72e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bc347c278c44c69444caf9ded72e1a already in progress: creating tablet
W20250625 07:52:23.263217 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet dddbc1200acf4e7fa93ef85b398d5832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dddbc1200acf4e7fa93ef85b398d5832 already in progress: creating tablet
I20250625 07:52:23.263224 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_602fb37488c14e66b4cfc551eb323c1c" } modify_external_catalogs: true
I20250625 07:52:23.263449 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_602fb37488c14e66b4cfc551eb323c1c" } modify_external_catalogs: true
W20250625 07:52:23.263770 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7480c07af7ce4a3889bd8d42d7419d57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7480c07af7ce4a3889bd8d42d7419d57 already in progress: creating tablet
I20250625 07:52:23.264151 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f" } modify_external_catalogs: true
I20250625 07:52:23.264312 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_54bc502b9aa347cfa3e69dcb4f2e8c8f" } modify_external_catalogs: true
W20250625 07:52:23.264403 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e10149f4ae0645ba88508ca51ae9f77a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e10149f4ae0645ba88508ca51ae9f77a already in progress: creating tablet
W20250625 07:52:23.264938 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c3d8b6b5c06248879e6052f6ae7fe46e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3d8b6b5c06248879e6052f6ae7fe46e already in progress: creating tablet
I20250625 07:52:23.268428 1162 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.270342 1162 tablet_bootstrap.cc:654] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.270329 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_abc25ebdf79c4455be53d3dfb090fdd6" } new_table_name: "default.Table_4564f3a341da4fa9964a50bb342bc684" modify_external_catalogs: true
I20250625 07:52:23.271725 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ba73f3114ea04e219b7a53b12922b3ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.272243 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba73f3114ea04e219b7a53b12922b3ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.272725 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]). Expected version 3 got 0
I20250625 07:52:23.274422 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a8adac4095f54fafa0b25109890670ec" } modify_external_catalogs: true
I20250625 07:52:23.274605 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a8adac4095f54fafa0b25109890670ec" } modify_external_catalogs: true
I20250625 07:52:23.276386 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]). Expected version 3 got 0
I20250625 07:52:23.277252 889 tablet_service.cc:1468] Processing CreateTablet for tablet f3027afac9d34fc9bdeb59bced467618 (DEFAULT_TABLE table=default.Table_c3c1fb488db2488da0ad1bf71b323567 [id=7c5cd44f2b4247caada06766ed7a6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.277671 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3027afac9d34fc9bdeb59bced467618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.280982 881 tablet_service.cc:1468] Processing CreateTablet for tablet e66a7cd87ce440a0a10da20cce2d82ba (DEFAULT_TABLE table=default.Table_b6fbe0d023e745a0a9cc130f361967d9 [id=be2d852f20a5452fb7d09c66bbe3ba9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.281354 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66a7cd87ce440a0a10da20cce2d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.284970 886 tablet_service.cc:1468] Processing CreateTablet for tablet e73689f413504e28ba97a83ab7b03fa1 (DEFAULT_TABLE table=default.Table_a87d4f6703a94cbb96962177f2bb0cd7 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.285312 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73689f413504e28ba97a83ab7b03fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.278697 1543 raft_consensus.cc:491] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.286378 1543 raft_consensus.cc:513] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.287271 1543 leader_election.cc:290] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.288951 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6b39c48d9948400a8f36e389cff0020a (DEFAULT_TABLE table=default.Table_f35dcb866b4e4dcc98712f798468a51e [id=5e24eebaf4784cec9d0bd20b0a8401e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.289273 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b39c48d9948400a8f36e389cff0020a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.289992 1543 raft_consensus.cc:491] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.290217 1543 raft_consensus.cc:513] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.290839 1543 leader_election.cc:290] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.292979 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9b66fad15471497390fc753b4015bb8f (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.293322 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b66fad15471497390fc753b4015bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.295141 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:23.295400 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:23.286163 1166 ts_tablet_manager.cc:1918] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.295850 1166 log.cc:1199] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2a9945f38ef148caac59acaadcb2ef4c
I20250625 07:52:23.296353 1166 ts_tablet_manager.cc:1939] T 2a9945f38ef148caac59acaadcb2ef4c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.297374 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:23.297576 912 raft_consensus.cc:2466] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:23.298012 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc (DEFAULT_TABLE table=default.Table_8441cf3142424cc78e9fd8690dd8752e [id=807e57cd425140d28a41226f07746546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.298243 884 tablet_service.cc:1468] Processing CreateTablet for tablet e529cb070cbf4272b900484ce268b291 (DEFAULT_TABLE table=default.Table_b3786a3e21ca46cdb5365fe9bc606a7f [id=e93f8f2f931e4956a6e4d64a31a49bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.298367 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.298557 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e529cb070cbf4272b900484ce268b291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.299305 1162 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.299471 1162 ts_tablet_manager.cc:1397] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250625 07:52:23.300554 1162 raft_consensus.cc:357] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.300820 1162 raft_consensus.cc:383] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.300921 1162 raft_consensus.cc:738] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.301164 1162 consensus_queue.cc:260] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.301759 713 leader_election.cc:343] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:23.301975 713 leader_election.cc:304] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.302810 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e54e9c8d68174135872e7b1d2d1086a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.303403 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e54e9c8d68174135872e7b1d2d1086a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.298012 895 tablet_service.cc:1468] Processing CreateTablet for tablet 11083173ebae45bc8c567cf5b750a06f (DEFAULT_TABLE table=default.Table_602fb37488c14e66b4cfc551eb323c1c [id=97644c90a0e2483f869ee95d6e661b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.304323 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11083173ebae45bc8c567cf5b750a06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.310496 1543 raft_consensus.cc:2802] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:23.310616 1543 raft_consensus.cc:491] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:23.310598 1651 raft_consensus.cc:491] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.310714 1543 raft_consensus.cc:3058] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.310715 1651 raft_consensus.cc:513] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:23.311304 1651 leader_election.cc:290] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.311920 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08bb47266e8b4a0fa5ec613560b8549e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:23.312485 851 leader_election.cc:343] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.314131 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_86dbdcd72c5a41f8adee048e24e3e9e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.314716 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86dbdcd72c5a41f8adee048e24e3e9e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.315413 1543 raft_consensus.cc:513] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.316044 1543 leader_election.cc:290] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.316989 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:23.317040 1162 ts_tablet_manager.cc:1428] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250625 07:52:23.316989 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8e8ec360a4e6476db25022ee41cf9eba" } new_table_name: "default.Table_6b88554443774ac1816a1e90b766e23d" modify_external_catalogs: true
I20250625 07:52:23.317139 912 raft_consensus.cc:3058] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.317780 1651 raft_consensus.cc:491] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.317893 1651 raft_consensus.cc:513] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.318434 1651 leader_election.cc:290] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.319020 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5a3751333bc48fc8a59b0e7b67a9935" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:23.319482 851 leader_election.cc:343] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.319749 912 raft_consensus.cc:2466] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:23.323935 713 leader_election.cc:304] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.324630 1543 raft_consensus.cc:2802] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:23.325114 883 tablet_service.cc:1468] Processing CreateTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f (DEFAULT_TABLE table=default.Table_fe9a633f1a8e450ba2a6db9adc198734 [id=34ea3bbfcd844cc5a790b1bab8001520]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.325479 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a1bfd87f084d50bd2b602b89c18a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.326500 1543 raft_consensus.cc:695] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:23.330968 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_152aaa7143664c5daca1d173d464e81e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.331704 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_152aaa7143664c5daca1d173d464e81e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.334501 1543 consensus_queue.cc:237] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.336133 1162 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.339049 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 2efffbfca58d44ffa97e56b5c10e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.339956 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5a3751333bc48fc8a59b0e7b67a9935" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.345044 1162 tablet_bootstrap.cc:654] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.346963 743 tablet_service.cc:1468] Processing CreateTablet for tablet f3027afac9d34fc9bdeb59bced467618 (DEFAULT_TABLE table=default.Table_c3c1fb488db2488da0ad1bf71b323567 [id=7c5cd44f2b4247caada06766ed7a6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.346992 758 tablet_service.cc:1468] Processing CreateTablet for tablet e66a7cd87ce440a0a10da20cce2d82ba (DEFAULT_TABLE table=default.Table_b6fbe0d023e745a0a9cc130f361967d9 [id=be2d852f20a5452fb7d09c66bbe3ba9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.347465 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3027afac9d34fc9bdeb59bced467618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.347873 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66a7cd87ce440a0a10da20cce2d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.348011 754 tablet_service.cc:1468] Processing CreateTablet for tablet e73689f413504e28ba97a83ab7b03fa1 (DEFAULT_TABLE table=default.Table_a87d4f6703a94cbb96962177f2bb0cd7 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.348474 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73689f413504e28ba97a83ab7b03fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.350836 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08bb47266e8b4a0fa5ec613560b8549e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.345909 1166 tablet_replica.cc:331] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:23.352896 850 leader_election.cc:343] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.353047 850 leader_election.cc:304] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.353317 1651 raft_consensus.cc:2747] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:23.353492 850 leader_election.cc:343] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.353595 850 leader_election.cc:304] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.353792 1651 raft_consensus.cc:2747] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.354955 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2a9945f38ef148caac59acaadcb2ef4c (table default.Table_4f645096238747a1a97bcd673b5438fd [id=836816eb909641daba87bdae20e0fd24]) successfully deleted
I20250625 07:52:23.356845 1660 raft_consensus.cc:491] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.360909 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6b39c48d9948400a8f36e389cff0020a (DEFAULT_TABLE table=default.Table_f35dcb866b4e4dcc98712f798468a51e [id=5e24eebaf4784cec9d0bd20b0a8401e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.362351 1160 tablet_replica.cc:331] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:23.362562 1160 raft_consensus.cc:2241] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.362772 1160 raft_consensus.cc:2270] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.365038 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 11083173ebae45bc8c567cf5b750a06f
I20250625 07:52:23.367020 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7db12504a1a54add920561dae27d2d2b
I20250625 07:52:23.367794 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f426b18fff7f499db91894562dc4736e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.368502 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f426b18fff7f499db91894562dc4736e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.370193 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 91a1d44dde9942a2bfd32a108b9dea92
I20250625 07:52:23.370355 1160 ts_tablet_manager.cc:1905] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.370918 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a1fac047060649b39adb2f5958d9816f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.371452 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1fac047060649b39adb2f5958d9816f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.376305 1166 raft_consensus.cc:2241] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.376399 1543 raft_consensus.cc:491] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.376554 1543 raft_consensus.cc:513] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.376675 1166 raft_consensus.cc:2270] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.376444 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dab6e9af194049cb8259ae333b704be3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.376752 742 tablet_service.cc:1468] Processing CreateTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f (DEFAULT_TABLE table=default.Table_fe9a633f1a8e450ba2a6db9adc198734 [id=34ea3bbfcd844cc5a790b1bab8001520]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.377183 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dab6e9af194049cb8259ae333b704be3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.377108 744 tablet_service.cc:1468] Processing CreateTablet for tablet e529cb070cbf4272b900484ce268b291 (DEFAULT_TABLE table=default.Table_b3786a3e21ca46cdb5365fe9bc606a7f [id=e93f8f2f931e4956a6e4d64a31a49bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.378413 1660 raft_consensus.cc:513] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.379222 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8b3050d2440e4b81907a06a8574cf631" } new_table_name: "default.Table_8cc8ba5db88541abb0f69c1df86ac4be" modify_external_catalogs: true
I20250625 07:52:23.379547 1660 leader_election.cc:290] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.379778 755 tablet_service.cc:1468] Processing CreateTablet for tablet 11083173ebae45bc8c567cf5b750a06f (DEFAULT_TABLE table=default.Table_602fb37488c14e66b4cfc551eb323c1c [id=97644c90a0e2483f869ee95d6e661b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.378747 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9b66fad15471497390fc753b4015bb8f (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.380472 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33466d4481c24294b5cc7a7736a1d737 consensus state has no leader
I20250625 07:52:23.380880 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ffc65594d4e14bd4947ee4f455719443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.381462 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffc65594d4e14bd4947ee4f455719443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.382969 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.383193 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc (DEFAULT_TABLE table=default.Table_8441cf3142424cc78e9fd8690dd8752e [id=807e57cd425140d28a41226f07746546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.383371 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:23.383543 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 11083173ebae45bc8c567cf5b750a06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.383582 912 raft_consensus.cc:2466] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:23.383693 1166 ts_tablet_manager.cc:1905] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.384289 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.386071 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.386438 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b39c48d9948400a8f36e389cff0020a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:23.387642 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bfa1c4d4b6e4530892118189a5ab516 consensus state has no leader
I20250625 07:52:23.389468 1543 leader_election.cc:290] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:23.389724 713 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b377c8111c9f4ac2ba05e65694d611b9
I20250625 07:52:23.390074 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a1bfd87f084d50bd2b602b89c18a2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:23.390167 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb98ab2850d746a8bf4733efbcc8bdbc consensus state has no leader
I20250625 07:52:23.390331 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.391278 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e529cb070cbf4272b900484ce268b291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.391304 713 leader_election.cc:304] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.391672 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11083173ebae45bc8c567cf5b750a06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.392508 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6e77ff1b8ca848ff8e1b4538645b4be2
I20250625 07:52:23.394491 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ee522622705d45f899ebec2ce759c046"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.394883 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b66fad15471497390fc753b4015bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.394934 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.396250 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e7ca753d58c84a17a3612fe36e374c7a" } new_table_name: "default.Table_7c0a9001bc454c42b4f7a4713a27dee9" modify_external_catalogs: true
W20250625 07:52:23.396651 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee522622705d45f899ebec2ce759c046 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:23.396944 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.397747 1160 ts_tablet_manager.cc:1918] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.397904 1160 log.cc:1199] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:23.398231 1160 ts_tablet_manager.cc:1939] T 250175f54ba94a9fbc2455d9cf27fb2e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:23.399497 1543 raft_consensus.cc:2802] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:23.399770 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_87e58d9e449a4f69bed998c1f045fbec" } modify_external_catalogs: true
I20250625 07:52:23.399924 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_87e58d9e449a4f69bed998c1f045fbec" } modify_external_catalogs: true
I20250625 07:52:23.400197 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_df524f8ba5984db8a7f578d01cd2bcd0" } new_table_name: "default.Table_6a60665ba1c74648904cfad10762f2c0" modify_external_catalogs: true
I20250625 07:52:23.402379 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.402760 1543 raft_consensus.cc:491] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:23.402878 1543 raft_consensus.cc:3058] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.410354 1651 raft_consensus.cc:491] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.410508 1651 raft_consensus.cc:513] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.411139 1651 leader_election.cc:290] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.338907 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 53c6c51a90754878977aeeaacf4773a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.413679 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7a2b94af59648239cfd1e5ce5e8b968" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.413991 1161 tablet_replica.cc:331] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:23.414161 1161 raft_consensus.cc:2241] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.414355 1161 raft_consensus.cc:2270] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.414880 1161 ts_tablet_manager.cc:1905] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.423177 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:23.423987 851 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.424546 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.426044 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4519b1fbf34ca8b33655f4d402aafc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:23.427341 1660 raft_consensus.cc:491] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.428025 1660 raft_consensus.cc:513] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.428226 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:23.428782 850 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.428903 850 leader_election.cc:304] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.429231 1651 raft_consensus.cc:2747] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.429996 1660 leader_election.cc:290] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.431180 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b377c8111c9f4ac2ba05e65694d611b9" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.431463 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:23.432437 713 leader_election.cc:343] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:23.439821 1161 ts_tablet_manager.cc:1918] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.440037 1161 log.cc:1199] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/53c6c51a90754878977aeeaacf4773a6
I20250625 07:52:23.440476 1161 ts_tablet_manager.cc:1939] T 53c6c51a90754878977aeeaacf4773a6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:23.447957 1543 raft_consensus.cc:513] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.449316 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:23.449534 912 raft_consensus.cc:3058] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.452351 912 raft_consensus.cc:2466] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:23.453274 713 leader_election.cc:304] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.453701 1660 raft_consensus.cc:2802] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:23.454289 1660 raft_consensus.cc:695] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:23.454672 1660 consensus_queue.cc:237] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.456333 1543 leader_election.cc:290] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.469627 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e73689f413504e28ba97a83ab7b03fa1 (DEFAULT_TABLE table=default.Table_a87d4f6703a94cbb96962177f2bb0cd7 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.470376 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f3027afac9d34fc9bdeb59bced467618 (DEFAULT_TABLE table=default.Table_c3c1fb488db2488da0ad1bf71b323567 [id=7c5cd44f2b4247caada06766ed7a6465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.471264 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 6b39c48d9948400a8f36e389cff0020a (DEFAULT_TABLE table=default.Table_f35dcb866b4e4dcc98712f798468a51e [id=5e24eebaf4784cec9d0bd20b0a8401e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.472061 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9b66fad15471497390fc753b4015bb8f (DEFAULT_TABLE table=default.Table_df524f8ba5984db8a7f578d01cd2bcd0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.474550 1028 tablet_service.cc:1468] Processing CreateTablet for tablet e66a7cd87ce440a0a10da20cce2d82ba (DEFAULT_TABLE table=default.Table_b6fbe0d023e745a0a9cc130f361967d9 [id=be2d852f20a5452fb7d09c66bbe3ba9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.479035 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc (DEFAULT_TABLE table=default.Table_8441cf3142424cc78e9fd8690dd8752e [id=807e57cd425140d28a41226f07746546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.484376 1166 ts_tablet_manager.cc:1918] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.484548 1166 log.cc:1199] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:23.484907 1166 ts_tablet_manager.cc:1939] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.485533 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c7a2b94af59648239cfd1e5ce5e8b968 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:23.486042 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 065822c50586488785f7d9d53841a4dd from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:23.487227 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 11083173ebae45bc8c567cf5b750a06f (DEFAULT_TABLE table=default.Table_602fb37488c14e66b4cfc551eb323c1c [id=97644c90a0e2483f869ee95d6e661b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.496891 1166 tablet_replica.cc:331] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.497123 1166 raft_consensus.cc:2241] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.497340 1166 raft_consensus.cc:2270] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.498135 1166 ts_tablet_manager.cc:1905] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.500253 1162 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:23.500252 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b66fad15471497390fc753b4015bb8f consensus state has no leader
I20250625 07:52:23.500433 1162 ts_tablet_manager.cc:1397] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.164s user 0.015s sys 0.039s
W20250625 07:52:23.501586 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5de0c5bf60b24808af124c95833d9c25 consensus state has no leader
I20250625 07:52:23.502334 1162 raft_consensus.cc:357] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.502543 1162 raft_consensus.cc:383] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.502617 1162 raft_consensus.cc:738] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.502707 1163 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:23.502856 1163 ts_tablet_manager.cc:1397] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.254s user 0.013s sys 0.014s
I20250625 07:52:23.502888 1162 consensus_queue.cc:260] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.503628 1162 ts_tablet_manager.cc:1428] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:23.505409 1163 raft_consensus.cc:357] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.505599 1163 raft_consensus.cc:383] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.505671 1163 raft_consensus.cc:738] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.505672 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f (DEFAULT_TABLE table=default.Table_fe9a633f1a8e450ba2a6db9adc198734 [id=34ea3bbfcd844cc5a790b1bab8001520]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.505861 1163 consensus_queue.cc:260] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.510447 1162 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.512252 1162 tablet_bootstrap.cc:654] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.513226 1166 ts_tablet_manager.cc:1918] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.513513 1166 log.cc:1199] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b377c8111c9f4ac2ba05e65694d611b9
I20250625 07:52:23.514802 1166 ts_tablet_manager.cc:1939] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.517613 1166 tablet_replica.cc:331] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.519214 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b377c8111c9f4ac2ba05e65694d611b9 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]) successfully deleted
I20250625 07:52:23.519766 1166 raft_consensus.cc:2241] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.520386 1166 raft_consensus.cc:2270] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.520956 1163 ts_tablet_manager.cc:1428] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250625 07:52:23.521623 1166 ts_tablet_manager.cc:1905] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.524570 589 catalog_manager.cc:5582] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:23.530949 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 065822c50586488785f7d9d53841a4dd
I20250625 07:52:23.531436 594 catalog_manager.cc:6652] ReplaceTablet: tablet 065822c50586488785f7d9d53841a4dd deleted and replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2
I20250625 07:52:23.531701 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 065822c50586488785f7d9d53841a4dd with delete_type TABLET_DATA_DELETED (replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.533989 746 tablet_service.cc:1468] Processing CreateTablet for tablet 20d6334831b942399273d8d1314d7908 (DEFAULT_TABLE table=default.Table_cdfa29007d814e609da9a03c52f558f2 [id=bdb4ab6163694d18a5d3a32f9a950fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.534584 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d6334831b942399273d8d1314d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.537169 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 90f2bef7baa54f1a96b3b59921d59457
W20250625 07:52:23.537381 713 leader_election.cc:343] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.537515 713 leader_election.cc:304] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:23.537746 713 raft_consensus.cc:2684] T c7a2b94af59648239cfd1e5ce5e8b968 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:23.538434 713 leader_election.cc:343] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:23.538745 713 leader_election.cc:343] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:23.539016 713 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.539141 713 leader_election.cc:304] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:23.539317 713 raft_consensus.cc:2684] T b377c8111c9f4ac2ba05e65694d611b9 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:23.538440 1162 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:23.539897 713 leader_election.cc:343] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.540014 1162 ts_tablet_manager.cc:1397] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.011s
I20250625 07:52:23.540645 1158 tablet_bootstrap.cc:492] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:23.540915 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.541077 1162 raft_consensus.cc:357] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.541265 1162 raft_consensus.cc:383] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.541358 1162 raft_consensus.cc:738] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.541431 1158 ts_tablet_manager.cc:1397] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.615s user 0.005s sys 0.006s
I20250625 07:52:23.541577 1162 consensus_queue.cc:260] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.542199 1162 ts_tablet_manager.cc:1428] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:23.542395 596 catalog_manager.cc:5582] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:23.542807 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73689f413504e28ba97a83ab7b03fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.542855 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]). Expected version 3 got 0
I20250625 07:52:23.543066 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
W20250625 07:52:23.546773 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5de0c5bf60b24808af124c95833d9c25 (table default.Table_7c0a9001bc454c42b4f7a4713a27dee9 [id=63d51352cf0b435ba49ef141c86bc32e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5de0c5bf60b24808af124c95833d9c25 consensus state has no leader
I20250625 07:52:23.547077 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.548638 1162 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.550469 1162 tablet_bootstrap.cc:654] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.553248 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a1bfd87f084d50bd2b602b89c18a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.555240 1651 raft_consensus.cc:491] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.555430 1651 raft_consensus.cc:513] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.556046 1651 leader_election.cc:290] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.556682 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29f2997d1d714100950383e6af04f315" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:23.557209 851 leader_election.cc:343] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.559616 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29f2997d1d714100950383e6af04f315" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.559607 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.560019 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbace0f99a44800bab1de5e38eb5112. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:23.560150 850 leader_election.cc:343] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.560160 753 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.560249 850 leader_election.cc:304] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.560600 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b39c48d9948400a8f36e389cff0020a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.563338 1651 raft_consensus.cc:2747] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.566026 1162 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.566622 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b66fad15471497390fc753b4015bb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.566712 1162 ts_tablet_manager.cc:1397] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250625 07:52:23.569208 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11083173ebae45bc8c567cf5b750a06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.569375 1162 raft_consensus.cc:357] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.569684 1162 raft_consensus.cc:383] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.569792 1162 raft_consensus.cc:738] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.570079 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66a7cd87ce440a0a10da20cce2d82ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.570102 1162 consensus_queue.cc:260] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.574873 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3910e10f6f4b41eeabfd11ac6c4343c7" } modify_external_catalogs: true
I20250625 07:52:23.575126 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3910e10f6f4b41eeabfd11ac6c4343c7" } modify_external_catalogs: true
I20250625 07:52:23.576231 1162 ts_tablet_manager.cc:1428] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 07:52:23.577631 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f801c447326646d1ab93ac87fff7843a" } modify_external_catalogs: true
I20250625 07:52:23.577764 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8adb7a38211942288ca93280cd05d4f5" } modify_external_catalogs: true
I20250625 07:52:23.577857 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f801c447326646d1ab93ac87fff7843a" } modify_external_catalogs: true
I20250625 07:52:23.577934 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8adb7a38211942288ca93280cd05d4f5" } modify_external_catalogs: true
I20250625 07:52:23.580677 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d3d7906832b2489497d12cae57a44328"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.581234 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3d7906832b2489497d12cae57a44328 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.581696 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_34ac7b0478f94f3d8a0c39d6671c95e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.582183 1162 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:23.582206 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34ac7b0478f94f3d8a0c39d6671c95e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.585036 742 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.585948 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3027afac9d34fc9bdeb59bced467618. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.587539 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_de8edff1052842d9bd91dc30913c757b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.588101 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de8edff1052842d9bd91dc30913c757b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.588807 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9bccda893fd5466886a4d253172309e1
I20250625 07:52:23.590232 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.591270 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cf47515a30a347929633a53b20a19b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.591528 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca8f9a686ffb42c18e83701403816a41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.591913 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf47515a30a347929633a53b20a19b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.592191 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_73f9bd402c6044089fc883b657511521" } new_table_name: "default.Table_ad97f0e793e84237beba40faefc5ec38" modify_external_catalogs: true
I20250625 07:52:23.592489 1162 tablet_bootstrap.cc:654] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:23.591945 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca8f9a686ffb42c18e83701403816a41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.598282 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bfb421ea711a461890e267153250acea
I20250625 07:52:23.599974 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a40dacaa394747aaa100a0527e14eed0
I20250625 07:52:23.600211 742 tablet_service.cc:1515] Processing DeleteTablet for tablet bfb421ea711a461890e267153250acea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.600780 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a40dacaa394747aaa100a0527e14eed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.601121 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5cbcfdbba5a84de1ac582e4687f347a5" } new_table_name: "default.Table_7fb5f2a41fbf4e11add0bf364fc3628e" modify_external_catalogs: true
I20250625 07:52:23.601498 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_58ba6295501d43f4b2c150eb6ac31aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.601972 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58ba6295501d43f4b2c150eb6ac31aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.603787 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_31f10020a22b49bc8778c19390b27fd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.604275 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f10020a22b49bc8778c19390b27fd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.605804 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c363caab5c8c4e61b584674a8c476ac7" } new_table_name: "default.Table_8f22051606e849db9f028420c2ac7979" modify_external_catalogs: true
I20250625 07:52:23.607208 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3a7a4153cc2f4ba0a19445f60f6703ca" } modify_external_catalogs: true
I20250625 07:52:23.607398 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3a7a4153cc2f4ba0a19445f60f6703ca" } modify_external_catalogs: true
I20250625 07:52:23.608306 1166 ts_tablet_manager.cc:1918] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.608492 1166 log.cc:1199] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/5c28a064006145eea000e74da10d0794
I20250625 07:52:23.608740 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4e3a4071789f48b5a7b5ab2c208f5130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.609310 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e3a4071789f48b5a7b5ab2c208f5130 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.614789 1166 ts_tablet_manager.cc:1939] T 5c28a064006145eea000e74da10d0794 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.616286 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 5c28a064006145eea000e74da10d0794 (table default.Table_9bbce7454a884554b31faffc79a33b99 [id=cbb5aafb8a9243d98c7d8578865c13b9]) successfully deleted
I20250625 07:52:23.618127 1166 tablet_replica.cc:331] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.618422 1166 raft_consensus.cc:2241] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.618647 1166 raft_consensus.cc:2270] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.619362 1166 ts_tablet_manager.cc:1905] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.625684 1162 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.625929 1162 ts_tablet_manager.cc:1397] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250625 07:52:23.626847 1162 raft_consensus.cc:357] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.627121 1162 raft_consensus.cc:383] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.627959 1162 raft_consensus.cc:738] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.628297 1162 consensus_queue.cc:260] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.632344 1162 ts_tablet_manager.cc:1428] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:23.633821 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_da75eea0e9054b62ac5c228ce0fa5b27 [id=b0f32fb69f33447db7620299affa88c8]). Expected version 2 got 0
I20250625 07:52:23.635931 1162 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.637888 1162 tablet_bootstrap.cc:654] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.637881 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_da75eea0e9054b62ac5c228ce0fa5b27 [id=b0f32fb69f33447db7620299affa88c8]). Expected version 2 got 0
I20250625 07:52:23.648854 1166 ts_tablet_manager.cc:1918] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.649165 1166 log.cc:1199] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6606a9559e6b47d28669083af5aee5bd
I20250625 07:52:23.649783 1166 ts_tablet_manager.cc:1939] T 6606a9559e6b47d28669083af5aee5bd P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.651516 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6606a9559e6b47d28669083af5aee5bd (table default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]) successfully deleted
I20250625 07:52:23.648916 1016 tablet_service.cc:1468] Processing CreateTablet for tablet e529cb070cbf4272b900484ce268b291 (DEFAULT_TABLE table=default.Table_b3786a3e21ca46cdb5365fe9bc606a7f [id=e93f8f2f931e4956a6e4d64a31a49bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.652175 1166 tablet_replica.cc:331] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.652288 1162 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.652451 1162 ts_tablet_manager.cc:1397] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20250625 07:52:23.652598 1166 raft_consensus.cc:2241] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.652848 1166 raft_consensus.cc:2270] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.653330 1162 raft_consensus.cc:357] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.653546 1162 raft_consensus.cc:383] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.653628 1162 raft_consensus.cc:738] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.653837 1166 ts_tablet_manager.cc:1905] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.653908 1162 consensus_queue.cc:260] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.657704 1162 ts_tablet_manager.cc:1428] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:23.661007 1162 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:23.664315 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6b4bca32de4a538071a652409e93b2 consensus state has no leader
I20250625 07:52:23.664661 1162 tablet_bootstrap.cc:654] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.666196 1166 ts_tablet_manager.cc:1918] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:23.666445 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b83f4c8544946d9adb3346df200b215 consensus state has no leader
I20250625 07:52:23.667791 1163 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:23.669625 1163 tablet_bootstrap.cc:654] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.673151 1166 log.cc:1199] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:23.673174 1162 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.673825 1162 ts_tablet_manager.cc:1397] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250625 07:52:23.673822 1166 ts_tablet_manager.cc:1939] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.675074 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c65f257ae653488a8b49b0b523d2cf82 (table default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]) successfully deleted
I20250625 07:52:23.675594 1166 tablet_replica.cc:331] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.675786 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9f655669b74b4a6594fbd63c45d15bf4
I20250625 07:52:23.675884 1166 raft_consensus.cc:2241] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.676817 1166 raft_consensus.cc:2270] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:23.677191 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62885e9218f64527854b79919e7725d5 consensus state has no leader
I20250625 07:52:23.677866 1166 ts_tablet_manager.cc:1905] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.679486 1160 tablet_replica.cc:331] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:23.679693 1160 raft_consensus.cc:2241] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.679849 1160 raft_consensus.cc:2270] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.680430 1160 ts_tablet_manager.cc:1905] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.680943 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.681178 1660 raft_consensus.cc:491] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.681311 1660 raft_consensus.cc:513] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.682040 1660 leader_election.cc:290] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.682569 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:23.682813 912 raft_consensus.cc:2466] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:23.683231 713 leader_election.cc:304] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.683542 1660 raft_consensus.cc:2802] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:23.683656 1660 raft_consensus.cc:491] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:23.683735 1660 raft_consensus.cc:3058] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.685272 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c65f257ae653488a8b49b0b523d2cf82" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.686712 1660 raft_consensus.cc:513] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.687657 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 206204us
I20250625 07:52:23.689798 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 250175f54ba94a9fbc2455d9cf27fb2e (table default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]) successfully deleted
I20250625 07:52:23.690382 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 11083173ebae45bc8c567cf5b750a06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.690586 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.690809 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.690956 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.691087 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.691255 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 065822c50586488785f7d9d53841a4dd with delete_type TABLET_DATA_DELETED (replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.690328 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
W20250625 07:52:23.691617 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2efffbfca58d44ffa97e56b5c10e9f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2efffbfca58d44ffa97e56b5c10e9f29 already in progress: creating tablet
I20250625 07:52:23.691736 883 tablet_service.cc:1468] Processing CreateTablet for tablet 20d6334831b942399273d8d1314d7908 (DEFAULT_TABLE table=default.Table_cdfa29007d814e609da9a03c52f558f2 [id=bdb4ab6163694d18a5d3a32f9a950fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.692219 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d6334831b942399273d8d1314d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.692548 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 53c6c51a90754878977aeeaacf4773a6 (table default.Table_658480d4343a409bad94a3ecbecb484b [id=bc06cb9b9fdc476c89010b80f89897da]) successfully deleted
I20250625 07:52:23.692723 1660 leader_election.cc:290] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.693342 1162 raft_consensus.cc:357] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.693522 1162 raft_consensus.cc:383] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.694005 1162 raft_consensus.cc:738] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.694314 1162 consensus_queue.cc:260] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.695632 884 tablet_service.cc:1515] Processing DeleteTablet for tablet bfb421ea711a461890e267153250acea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.695806 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a40dacaa394747aaa100a0527e14eed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.695606 1158 raft_consensus.cc:357] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:23.695969 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.696110 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.691778 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.696527 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbace0f99a44800bab1de5e38eb5112. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.696633 1162 ts_tablet_manager.cc:1428] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250625 07:52:23.700021 1162 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.701903 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_212082b707ae4f5689a2dcffa943a19c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.702175 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.702796 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a8813cc630ad48ca87a8bcc192010644" } new_table_name: "default.Table_2207905ba3a74490b97826054d872e6c" modify_external_catalogs: true
I20250625 07:52:23.703179 1162 tablet_bootstrap.cc:654] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.704243 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_377ef3ec98894b328b266b665f9a75f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.705009 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_377ef3ec98894b328b266b665f9a75f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:23.706529 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_212082b707ae4f5689a2dcffa943a19c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.709040 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9" } new_table_name: "default.Table_e2bd1f5414044ed5bc39851226d92171" modify_external_catalogs: true
I20250625 07:52:23.709450 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e529cb070cbf4272b900484ce268b291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.710357 912 raft_consensus.cc:3058] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.711566 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edda0c3f60f64ed0b965eca52c5b3591" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:23.712435 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.713222 912 raft_consensus.cc:2466] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:23.713775 713 leader_election.cc:304] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.714231 1660 raft_consensus.cc:2802] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:23.714413 1660 raft_consensus.cc:695] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:23.714784 1660 consensus_queue.cc:237] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.718066 713 leader_election.cc:343] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.718236 713 leader_election.cc:304] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:23.718475 713 raft_consensus.cc:2684] T c65f257ae653488a8b49b0b523d2cf82 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:23.718643 1163 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:23.718776 1163 ts_tablet_manager.cc:1397] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.006s
I20250625 07:52:23.719573 1163 raft_consensus.cc:357] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.719791 1163 raft_consensus.cc:383] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.719870 1163 raft_consensus.cc:738] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.720129 1163 consensus_queue.cc:260] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.720366 713 leader_election.cc:343] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.720829 1651 raft_consensus.cc:491] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.721036 1651 raft_consensus.cc:513] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.721858 1651 leader_election.cc:290] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.723050 1166 ts_tablet_manager.cc:1918] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.723109 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:23.723192 1166 log.cc:1199] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/81976db0a2f94128a89842a12f067231
I20250625 07:52:23.723308 775 raft_consensus.cc:2466] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:23.723595 1166 ts_tablet_manager.cc:1939] T 81976db0a2f94128a89842a12f067231 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.723814 851 leader_election.cc:304] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:23.726040 1651 raft_consensus.cc:2802] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:23.726178 1651 raft_consensus.cc:491] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:23.726264 1651 raft_consensus.cc:3058] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:23.729048 713 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.730201 1166 tablet_replica.cc:331] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.730417 1166 raft_consensus.cc:2241] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.733525 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 81976db0a2f94128a89842a12f067231 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
W20250625 07:52:23.734082 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 11083173ebae45bc8c567cf5b750a06f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11083173ebae45bc8c567cf5b750a06f already in progress: creating tablet
I20250625 07:52:23.734465 1166 raft_consensus.cc:2270] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:23.735271 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 91a1d44dde9942a2bfd32a108b9dea92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a1d44dde9942a2bfd32a108b9dea92 already in progress: creating tablet
W20250625 07:52:23.735551 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7db12504a1a54add920561dae27d2d2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db12504a1a54add920561dae27d2d2b already in progress: creating tablet
I20250625 07:52:23.736275 1651 raft_consensus.cc:513] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.736608 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e77ff1b8ca848ff8e1b4538645b4be2 already in progress: creating tablet
W20250625 07:52:23.736861 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 065822c50586488785f7d9d53841a4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 065822c50586488785f7d9d53841a4dd already in progress: creating tablet
W20250625 07:52:23.737123 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 90f2bef7baa54f1a96b3b59921d59457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90f2bef7baa54f1a96b3b59921d59457 already in progress: creating tablet
I20250625 07:52:23.737227 1166 ts_tablet_manager.cc:1905] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.740525 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:23.740708 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_eb85d0d01edc400eabcdd8a8175dfbf9" } modify_external_catalogs: true
I20250625 07:52:23.740932 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_eb85d0d01edc400eabcdd8a8175dfbf9" } modify_external_catalogs: true
I20250625 07:52:23.740983 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]). Expected version 2 got 0
I20250625 07:52:23.744232 1166 ts_tablet_manager.cc:1918] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.744378 1166 log.cc:1199] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a777f64425384be5bccc86b197626618
I20250625 07:52:23.738778 1158 raft_consensus.cc:383] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.745723 1163 ts_tablet_manager.cc:1428] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.027s user 0.004s sys 0.005s
I20250625 07:52:23.745585 591 catalog_manager.cc:5582] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:23.745786 746 tablet_service.cc:1468] Processing CreateTablet for tablet 97905f211b7141a3948988eab63b5994 (DEFAULT_TABLE table=default.Table_ee522622705d45f899ebec2ce759c046 [id=24d5c7312ade4ab691c8025842e7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.745810 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "875fa1b46903452eaed1f2235d63dfdf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:23.745981 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]). Expected version 2 got 0
I20250625 07:52:23.746065 1163 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:23.746160 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97905f211b7141a3948988eab63b5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.746156 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:23.749173 1158 raft_consensus.cc:738] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.749222 893 tablet_service.cc:1468] Processing CreateTablet for tablet 97905f211b7141a3948988eab63b5994 (DEFAULT_TABLE table=default.Table_ee522622705d45f899ebec2ce759c046 [id=24d5c7312ade4ab691c8025842e7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.749437 1158 consensus_queue.cc:260] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:23.749645 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97905f211b7141a3948988eab63b5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.752252 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3bff632e68674316bd19f3829ce55ae4 (DEFAULT_TABLE table=default.Table_152aaa7143664c5daca1d173d464e81e [id=4a2f0b3512a247c9829ee07f27486909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.752643 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff632e68674316bd19f3829ce55ae4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:23.753894 850 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.754186 1166 ts_tablet_manager.cc:1939] T a777f64425384be5bccc86b197626618 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.755441 883 tablet_service.cc:1468] Processing CreateTablet for tablet 3bff632e68674316bd19f3829ce55ae4 (DEFAULT_TABLE table=default.Table_152aaa7143664c5daca1d173d464e81e [id=4a2f0b3512a247c9829ee07f27486909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.755678 891 tablet_service.cc:1468] Processing CreateTablet for tablet f57666c394834fc49f287258e45f741b (DEFAULT_TABLE table=default.Table_ffc65594d4e14bd4947ee4f455719443 [id=d3879ff42fd442ad9e37a6f45f9270af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.755846 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff632e68674316bd19f3829ce55ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.756029 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57666c394834fc49f287258e45f741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.765014 886 tablet_service.cc:1468] Processing CreateTablet for tablet 48573933194547a0810fd1742294d525 (DEFAULT_TABLE table=default.Table_dab6e9af194049cb8259ae333b704be3 [id=ff0976f23c3b458aad16d7c5d45a1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.765388 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:23.765460 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48573933194547a0810fd1742294d525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.769071 885 tablet_service.cc:1468] Processing CreateTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.769299 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.769618 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a127ebc7a74274b60cdd7d08ce22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.770008 1166 tablet_replica.cc:331] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:23.770215 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 20d6334831b942399273d8d1314d7908 (DEFAULT_TABLE table=default.Table_cdfa29007d814e609da9a03c52f558f2 [id=bdb4ab6163694d18a5d3a32f9a950fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.770272 1166 raft_consensus.cc:2241] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:23.770570 1166 raft_consensus.cc:2270] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:23.770895 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 065822c50586488785f7d9d53841a4dd with delete_type TABLET_DATA_DELETED (replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.771168 1651 leader_election.cc:290] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.771790 1166 ts_tablet_manager.cc:1905] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:23.773180 749 tablet_service.cc:1468] Processing CreateTablet for tablet f57666c394834fc49f287258e45f741b (DEFAULT_TABLE table=default.Table_ffc65594d4e14bd4947ee4f455719443 [id=d3879ff42fd442ad9e37a6f45f9270af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.773674 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57666c394834fc49f287258e45f741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.775059 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 91a1d44dde9942a2bfd32a108b9dea92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.775717 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.776335 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 11083173ebae45bc8c567cf5b750a06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.777376 744 tablet_service.cc:1468] Processing CreateTablet for tablet 48573933194547a0810fd1742294d525 (DEFAULT_TABLE table=default.Table_dab6e9af194049cb8259ae333b704be3 [id=ff0976f23c3b458aad16d7c5d45a1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.777199 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.777871 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e (DEFAULT_TABLE table=default.Table_a1fac047060649b39adb2f5958d9816f [id=770d50d279c446d1855dd08a23089387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.779233 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:23.779889 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:23.780718 850 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.781353 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.781585 883 tablet_service.cc:1468] Processing CreateTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e (DEFAULT_TABLE table=default.Table_a1fac047060649b39adb2f5958d9816f [id=770d50d279c446d1855dd08a23089387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.782083 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f09c9fd1b3a4dae9c8b366000737b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.785441 884 tablet_service.cc:1468] Processing CreateTablet for tablet babaf4dbc59147699c7b9a16c82e0774 (DEFAULT_TABLE table=default.Table_86dbdcd72c5a41f8adee048e24e3e9e8 [id=24f91349533c4e498db5d34e7015a54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.785964 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet babaf4dbc59147699c7b9a16c82e0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.786674 1651 raft_consensus.cc:491] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:23.786809 1651 raft_consensus.cc:513] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.787477 1651 leader_election.cc:290] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:23.789404 881 tablet_service.cc:1468] Processing CreateTablet for tablet 28dddfa84bc1497a962c718dad7d3b6b (DEFAULT_TABLE table=default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9 [id=99e48152947545fcaccf01cea86c3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.789835 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dddfa84bc1497a962c718dad7d3b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.789781 895 tablet_service.cc:1468] Processing CreateTablet for tablet 267f9accd9b34154aa0896055a930c73 (DEFAULT_TABLE table=default.Table_ba73f3114ea04e219b7a53b12922b3ea [id=fdd3d4b230684386a010ad62e0a2bc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.790104 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267f9accd9b34154aa0896055a930c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.790423 889 tablet_service.cc:1468] Processing CreateTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.790753 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ef840a3ed40bd939b3fe62df3cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.793119 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db9e8d0fefa47bfaeee462dba096b49" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:23.798898 850 leader_election.cc:343] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.800577 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bcd09e33bddc4315879e476a6897ba21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.801294 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcd09e33bddc4315879e476a6897ba21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.802624 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb831997b8cf4b99a7779a4a6b23a238" } modify_external_catalogs: true
I20250625 07:52:23.802836 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb831997b8cf4b99a7779a4a6b23a238" } modify_external_catalogs: true
I20250625 07:52:23.805155 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_35e64151eae847128b84b16f6c0ab3ea" } modify_external_catalogs: true
I20250625 07:52:23.805377 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_35e64151eae847128b84b16f6c0ab3ea" } modify_external_catalogs: true
I20250625 07:52:23.814610 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d6334831b942399273d8d1314d7908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.814694 893 tablet_service.cc:1468] Processing CreateTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 (DEFAULT_TABLE table=default.Table_e54e9c8d68174135872e7b1d2d1086a4 [id=5796e4a299044a67994e6f4d851bdcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.815135 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f803ed26a94ee081bd9a7c7eb1d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.818272 743 tablet_service.cc:1468] Processing CreateTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 (DEFAULT_TABLE table=default.Table_e54e9c8d68174135872e7b1d2d1086a4 [id=5796e4a299044a67994e6f4d851bdcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.821285 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41e53efb6f342d2b1e4d642f7aaad10" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:23.821496 775 raft_consensus.cc:3058] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:23.821794 758 tablet_service.cc:1468] Processing CreateTablet for tablet babaf4dbc59147699c7b9a16c82e0774 (DEFAULT_TABLE table=default.Table_86dbdcd72c5a41f8adee048e24e3e9e8 [id=24f91349533c4e498db5d34e7015a54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.821125 754 tablet_service.cc:1468] Processing CreateTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.822958 1162 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.823210 747 tablet_service.cc:1468] Processing CreateTablet for tablet 28dddfa84bc1497a962c718dad7d3b6b (DEFAULT_TABLE table=default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9 [id=99e48152947545fcaccf01cea86c3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.823487 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48573933194547a0810fd1742294d525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.824538 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a777f64425384be5bccc86b197626618 (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
I20250625 07:52:23.824983 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f09c9fd1b3a4dae9c8b366000737b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.826503 1162 ts_tablet_manager.cc:1397] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.002s
I20250625 07:52:23.827394 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f803ed26a94ee081bd9a7c7eb1d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.827203 750 tablet_service.cc:1468] Processing CreateTablet for tablet 267f9accd9b34154aa0896055a930c73 (DEFAULT_TABLE table=default.Table_ba73f3114ea04e219b7a53b12922b3ea [id=fdd3d4b230684386a010ad62e0a2bc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.828135 751 tablet_service.cc:1468] Processing CreateTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.827872 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db9e8d0fefa47bfaeee462dba096b49" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:23.834470 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ef840a3ed40bd939b3fe62df3cda2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:23.835292 851 leader_election.cc:343] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:23.835433 851 leader_election.cc:304] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.835796 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267f9accd9b34154aa0896055a930c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.836473 1651 raft_consensus.cc:2747] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:23.837495 1158 ts_tablet_manager.cc:1428] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.296s user 0.002s sys 0.001s
I20250625 07:52:23.838472 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dddfa84bc1497a962c718dad7d3b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.840451 1158 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:23.840790 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet babaf4dbc59147699c7b9a16c82e0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.841732 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a127ebc7a74274b60cdd7d08ce22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.845049 1166 ts_tablet_manager.cc:1918] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:23.845165 1166 log.cc:1199] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:23.845518 1166 ts_tablet_manager.cc:1939] T e7100ef72ff14c24be92c21c6eb8a9de P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:23.845983 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet bfb421ea711a461890e267153250acea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.847998 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3bff632e68674316bd19f3829ce55ae4 (DEFAULT_TABLE table=default.Table_152aaa7143664c5daca1d173d464e81e [id=4a2f0b3512a247c9829ee07f27486909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.848769 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 97905f211b7141a3948988eab63b5994 (DEFAULT_TABLE table=default.Table_ee522622705d45f899ebec2ce759c046 [id=24d5c7312ade4ab691c8025842e7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.849211 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet a40dacaa394747aaa100a0527e14eed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.849800 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f655669b74b4a6594fbd63c45d15bf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:23.851631 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28dddfa84bc1497a962c718dad7d3b6b consensus state has no leader
I20250625 07:52:23.852161 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_497900c260fb4dbaa34bbeb002b540c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.852867 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_497900c260fb4dbaa34bbeb002b540c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.853938 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9ede02c06bf540edbff83b6d2e68d206" } modify_external_catalogs: true
I20250625 07:52:23.854116 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9ede02c06bf540edbff83b6d2e68d206" } modify_external_catalogs: true
W20250625 07:52:23.854924 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 328b21f669c442fb86259500e794d222 consensus state has no leader
I20250625 07:52:23.852200 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7a175cc57ae741328a3d515859d7a98a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.855849 1162 raft_consensus.cc:357] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:23.856035 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a175cc57ae741328a3d515859d7a98a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.856061 1162 raft_consensus.cc:383] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.856137 1162 raft_consensus.cc:738] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.856419 1162 consensus_queue.cc:260] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.857002 1162 ts_tablet_manager.cc:1428] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20250625 07:52:23.857585 1162 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.859572 1162 tablet_bootstrap.cc:654] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.866305 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8a130964fd79498ba4aad19cbd88c820" } new_table_name: "default.Table_1f8502fb577748268f87ac63e15ec4f5" modify_external_catalogs: true
I20250625 07:52:23.869180 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3666ef79a51b4d33a802675181e67fac" } new_table_name: "default.Table_7cf5acf3a4244f19a9151d898f7c7e5a" modify_external_catalogs: true
I20250625 07:52:23.870924 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e (DEFAULT_TABLE table=default.Table_a1fac047060649b39adb2f5958d9816f [id=770d50d279c446d1855dd08a23089387]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.871649 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.872371 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 48573933194547a0810fd1742294d525 (DEFAULT_TABLE table=default.Table_dab6e9af194049cb8259ae333b704be3 [id=ff0976f23c3b458aad16d7c5d45a1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.873008 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f57666c394834fc49f287258e45f741b (DEFAULT_TABLE table=default.Table_ffc65594d4e14bd4947ee4f455719443 [id=d3879ff42fd442ad9e37a6f45f9270af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.885133 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e7100ef72ff14c24be92c21c6eb8a9de (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
W20250625 07:52:23.885416 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9bccda893fd5466886a4d253172309e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bccda893fd5466886a4d253172309e1 already in progress: creating tablet
W20250625 07:52:23.885598 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bfb421ea711a461890e267153250acea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb421ea711a461890e267153250acea already in progress: creating tablet
I20250625 07:52:23.885680 1163 tablet_bootstrap.cc:654] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:23.885748 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a40dacaa394747aaa100a0527e14eed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40dacaa394747aaa100a0527e14eed0 already in progress: creating tablet
W20250625 07:52:23.885939 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9f655669b74b4a6594fbd63c45d15bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f655669b74b4a6594fbd63c45d15bf4 already in progress: creating tablet
I20250625 07:52:23.898811 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 (DEFAULT_TABLE table=default.Table_e54e9c8d68174135872e7b1d2d1086a4 [id=5796e4a299044a67994e6f4d851bdcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.898957 775 raft_consensus.cc:2466] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:23.899462 851 leader_election.cc:304] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:23.899907 1651 raft_consensus.cc:2802] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:23.900022 1651 raft_consensus.cc:695] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:23.900244 1651 consensus_queue.cc:237] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.913594 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:23.915517 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.917618 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca3d0678d40245259f27ff872ef5ac3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:23.918149 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fcc3afc210a540058294121d2b83c66d
W20250625 07:52:23.918262 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca3d0678d40245259f27ff872ef5ac3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.918926 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a37bdd5dd7c1405ea495ec1a66f47133" } modify_external_catalogs: true
I20250625 07:52:23.919045 886 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:23.919099 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a37bdd5dd7c1405ea495ec1a66f47133" } modify_external_catalogs: true
I20250625 07:52:23.919598 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4a167c7e87544c0ea7c10b4496629916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:23.920111 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a167c7e87544c0ea7c10b4496629916 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:23.927536 1162 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.927708 1162 ts_tablet_manager.cc:1397] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250625 07:52:23.905130 1019 tablet_service.cc:1468] Processing CreateTablet for tablet babaf4dbc59147699c7b9a16c82e0774 (DEFAULT_TABLE table=default.Table_86dbdcd72c5a41f8adee048e24e3e9e8 [id=24f91349533c4e498db5d34e7015a54b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.910871 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbace0f99a44800bab1de5e38eb5112. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.922797 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 28dddfa84bc1497a962c718dad7d3b6b (DEFAULT_TABLE table=default.Table_80ab22bfbebb4c4abce0ca4827b9fcd9 [id=99e48152947545fcaccf01cea86c3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.929595 1015 tablet_service.cc:1468] Processing CreateTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 (DEFAULT_TABLE table=default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:23.931290 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f51ef840a3ed40bd939b3fe62df3cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.932878 912 raft_consensus.cc:1273] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:23.934824 751 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:23.934824 1660 consensus_queue.cc:1035] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 07:52:23.935110 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28dddfa84bc1497a962c718dad7d3b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.935384 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:23.936769 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fcc3afc210a540058294121d2b83c66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcc3afc210a540058294121d2b83c66d already in progress: creating tablet
W20250625 07:52:23.937206 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 07a07af928ff4d3d894b074ef0cf44eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a07af928ff4d3d894b074ef0cf44eb already in progress: creating tablet
I20250625 07:52:23.946712 1162 raft_consensus.cc:357] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.946931 1162 raft_consensus.cc:383] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.947037 1162 raft_consensus.cc:738] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.947515 1162 consensus_queue.cc:260] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.948108 1162 ts_tablet_manager.cc:1428] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 07:52:23.948743 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet babaf4dbc59147699c7b9a16c82e0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.952622 1162 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.954604 1162 tablet_bootstrap.cc:654] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.958580 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bff632e68674316bd19f3829ce55ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.964324 1158 tablet_bootstrap.cc:654] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.964602 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f09c9fd1b3a4dae9c8b366000737b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.969058 1162 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:23.969267 1162 ts_tablet_manager.cc:1397] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20250625 07:52:23.969511 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97905f211b7141a3948988eab63b5994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.970078 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.970198 1162 raft_consensus.cc:357] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.970642 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a127ebc7a74274b60cdd7d08ce22cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.970829 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:23.970935 1162 raft_consensus.cc:383] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:23.971037 1162 raft_consensus.cc:738] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:23.971316 1162 consensus_queue.cc:260] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:23.971889 1162 ts_tablet_manager.cc:1428] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250625 07:52:23.972250 1162 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:23.978338 1162 tablet_bootstrap.cc:654] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:23.980754 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f57666c394834fc49f287258e45f741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.988420 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48573933194547a0810fd1742294d525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:23.991007 1543 consensus_queue.cc:1035] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:52:24.006105 713 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.006456 1160 ts_tablet_manager.cc:1918] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.006588 1160 log.cc:1199] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:24.006414 1716 tablet.cc:1722] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:24.006877 1160 ts_tablet_manager.cc:1939] T a5a3751333bc48fc8a59b0e7b67a9935 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:24.007716 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e77ff1b8ca848ff8e1b4538645b4be2 already in progress: creating tablet
I20250625 07:52:24.007998 1160 tablet_replica.cc:331] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.008186 1160 raft_consensus.cc:2241] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.008208 1717 tablet.cc:1722] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:24.008387 1160 raft_consensus.cc:2270] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.008387 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a5a3751333bc48fc8a59b0e7b67a9935 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
W20250625 07:52:24.008664 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2efffbfca58d44ffa97e56b5c10e9f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2efffbfca58d44ffa97e56b5c10e9f29 already in progress: creating tablet
I20250625 07:52:24.010136 1163 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.010272 1163 ts_tablet_manager.cc:1397] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.264s user 0.005s sys 0.012s
I20250625 07:52:24.010987 1163 raft_consensus.cc:357] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.011168 1163 raft_consensus.cc:383] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.011221 1163 raft_consensus.cc:738] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.011418 1163 consensus_queue.cc:260] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.012401 1160 ts_tablet_manager.cc:1905] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.013813 1163 ts_tablet_manager.cc:1428] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:24.014081 1163 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:24.017225 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bfcef152c9844a59989c532bd60c07d consensus state has no leader
I20250625 07:52:24.017168 593 catalog_manager.cc:5582] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:24.017640 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
I20250625 07:52:24.017601 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0fd3c55764824ab883ec46fb92e6fdee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.018168 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fd3c55764824ab883ec46fb92e6fdee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.019018 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8df215ea93ca49c391a3c3526be6fc22" } new_table_name: "default.Table_0a285a3966f14deb8b22b7f4eacd7e71" modify_external_catalogs: true
W20250625 07:52:24.019414 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07837660a7db47299b8e11f3335cb5d7 consensus state has no leader
I20250625 07:52:24.021122 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f678232d9ed444da9031bda1a4bf561e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.021734 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f678232d9ed444da9031bda1a4bf561e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.023432 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e67edafbb8384a69ab2a92fd5fabd83f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.023631 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d2c88638ac1843649fa7060cf1c62f90
W20250625 07:52:24.023989 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e67edafbb8384a69ab2a92fd5fabd83f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.024343 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 84f47caf3bae41b79b0aa1e320f15da4 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:24.025683 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 184cd64146f141db8e870af121a7772f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:24.027243 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f6fbe5526772435eb6ff141715994b65
I20250625 07:52:24.032096 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 875fa1b46903452eaed1f2235d63dfdf
I20250625 07:52:24.032392 1162 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.032852 1162 ts_tablet_manager.cc:1397] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.003s
I20250625 07:52:24.034251 1162 raft_consensus.cc:357] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.034582 1162 raft_consensus.cc:383] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.034706 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f6fbe5526772435eb6ff141715994b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.034763 1162 raft_consensus.cc:738] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.034917 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 875fa1b46903452eaed1f2235d63dfdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.034999 1162 consensus_queue.cc:260] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.035451 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d2c88638ac1843649fa7060cf1c62f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.037036 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d2c88638ac1843649fa7060cf1c62f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.037570 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 875fa1b46903452eaed1f2235d63dfdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.037371 1162 ts_tablet_manager.cc:1428] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:24.037801 1166 tablet_replica.cc:331] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.037887 747 tablet_service.cc:1515] Processing DeleteTablet for tablet f6fbe5526772435eb6ff141715994b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.037938 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.038098 1166 raft_consensus.cc:2241] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:24.038395 1166 pending_rounds.cc:70] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:24.038473 1166 pending_rounds.cc:77] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432217458843648 op_type: NO_OP noop_request { }
I20250625 07:52:24.038590 1166 raft_consensus.cc:2887] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:24.038714 1166 raft_consensus.cc:2270] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.040238 1166 ts_tablet_manager.cc:1905] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:24.040938 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d2c88638ac1843649fa7060cf1c62f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c88638ac1843649fa7060cf1c62f90 already in progress: creating tablet
I20250625 07:52:24.050591 1166 ts_tablet_manager.cc:1918] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250625 07:52:24.053970 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fed43adfae594e3b91ed1012838c3317 consensus state has no leader
I20250625 07:52:24.055750 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f803ed26a94ee081bd9a7c7eb1d660. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.057391 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3446d001da73487c8634225bf869b5a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.057744 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d3d7906832b2489497d12cae57a44328" } new_table_name: "default.Table_3d40c073684a48a69c3c60f46aab48bc" modify_external_catalogs: true
W20250625 07:52:24.058113 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3446d001da73487c8634225bf869b5a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.059228 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_15c9f1d9e9e842ef8ce4b125b9350af3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.059476 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5d201449036c4c1a8351c46d645de96c" } new_table_name: "default.Table_96d96b9fee4d48efaeb2f22ff67b43e6" modify_external_catalogs: true
W20250625 07:52:24.059823 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15c9f1d9e9e842ef8ce4b125b9350af3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.063033 1166 log.cc:1199] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/875fa1b46903452eaed1f2235d63dfdf
I20250625 07:52:24.063630 1166 ts_tablet_manager.cc:1939] T 875fa1b46903452eaed1f2235d63dfdf P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.062538 1163 tablet_bootstrap.cc:654] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.067821 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:24.072396 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:24.062891 1162 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.077069 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]) successfully deleted
W20250625 07:52:24.077972 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f6fbe5526772435eb6ff141715994b65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fbe5526772435eb6ff141715994b65 already in progress: creating tablet
I20250625 07:52:24.078585 1162 tablet_bootstrap.cc:654] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.079707 1166 tablet_replica.cc:331] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.080873 1166 raft_consensus.cc:2241] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.081123 1166 raft_consensus.cc:2270] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.086119 1162 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.086354 1162 ts_tablet_manager.cc:1397] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20250625 07:52:24.086616 1163 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.086828 1163 ts_tablet_manager.cc:1397] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.003s
I20250625 07:52:24.087693 1162 raft_consensus.cc:357] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.087895 1162 raft_consensus.cc:383] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.087976 1162 raft_consensus.cc:738] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.088223 1162 consensus_queue.cc:260] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.088654 1162 ts_tablet_manager.cc:1428] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:24.089025 1162 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:24.090703 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 065822c50586488785f7d9d53841a4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 065822c50586488785f7d9d53841a4dd already in progress: creating tablet
I20250625 07:52:24.090919 1162 tablet_bootstrap.cc:654] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:24.091061 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 91a1d44dde9942a2bfd32a108b9dea92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a1d44dde9942a2bfd32a108b9dea92 already in progress: creating tablet
I20250625 07:52:24.092973 886 tablet_service.cc:1468] Processing CreateTablet for tablet 7d42f5f3910742f0942742eb77ea1435 (DEFAULT_TABLE table=default.Table_58ba6295501d43f4b2c150eb6ac31aa2 [id=e78e1f8faa4c4ac5b908fca952514a41]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.093288 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1983df4902f48d2b408a99b440368e7 consensus state has no leader
I20250625 07:52:24.093400 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d42f5f3910742f0942742eb77ea1435. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.093783 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a30941e3b9c475db17f961c38d96ef4 consensus state has no leader
I20250625 07:52:24.094995 747 tablet_service.cc:1468] Processing CreateTablet for tablet 7d42f5f3910742f0942742eb77ea1435 (DEFAULT_TABLE table=default.Table_58ba6295501d43f4b2c150eb6ac31aa2 [id=e78e1f8faa4c4ac5b908fca952514a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.095429 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d42f5f3910742f0942742eb77ea1435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.101051 1162 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.101292 1162 ts_tablet_manager.cc:1397] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20250625 07:52:24.101611 744 tablet_service.cc:1468] Processing CreateTablet for tablet a02593ec05be42989afc7e2487cfb7e0 (DEFAULT_TABLE table=default.Table_4e3a4071789f48b5a7b5ab2c208f5130 [id=ae6dca9d3beb488aa00a15d2307bd0d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.102067 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02593ec05be42989afc7e2487cfb7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.102517 1162 raft_consensus.cc:357] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.102883 1162 raft_consensus.cc:383] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.103035 1162 raft_consensus.cc:738] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.103367 1162 consensus_queue.cc:260] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.104097 1162 ts_tablet_manager.cc:1428] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:24.101236 886 tablet_service.cc:1468] Processing CreateTablet for tablet a02593ec05be42989afc7e2487cfb7e0 (DEFAULT_TABLE table=default.Table_4e3a4071789f48b5a7b5ab2c208f5130 [id=ae6dca9d3beb488aa00a15d2307bd0d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.109261 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02593ec05be42989afc7e2487cfb7e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.110991 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7db12504a1a54add920561dae27d2d2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db12504a1a54add920561dae27d2d2b already in progress: creating tablet
I20250625 07:52:24.112365 1162 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.113440 1163 raft_consensus.cc:357] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.113638 1163 raft_consensus.cc:383] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.113709 1163 raft_consensus.cc:738] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.113909 1163 consensus_queue.cc:260] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.114373 1163 ts_tablet_manager.cc:1428] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250625 07:52:24.114418 1162 tablet_bootstrap.cc:654] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.114872 1163 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.116467 1163 tablet_bootstrap.cc:654] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.117326 1166 ts_tablet_manager.cc:1905] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.117900 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0c7a05e580684fb2811d89a355b0a26e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.118500 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c7a05e580684fb2811d89a355b0a26e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:24.121202 583 catalog_manager.cc:4895] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 11083173ebae45bc8c567cf5b750a06f because the tablet was not found. No further retry: Not found: Tablet not found: 11083173ebae45bc8c567cf5b750a06f
W20250625 07:52:24.121719 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 90f2bef7baa54f1a96b3b59921d59457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90f2bef7baa54f1a96b3b59921d59457 already in progress: creating tablet
I20250625 07:52:24.121788 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2d792b2565234096a0bc44a9c82abc1c" } modify_external_catalogs: true
I20250625 07:52:24.121950 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2d792b2565234096a0bc44a9c82abc1c" } modify_external_catalogs: true
W20250625 07:52:24.122140 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9bccda893fd5466886a4d253172309e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bccda893fd5466886a4d253172309e1 already in progress: creating tablet
I20250625 07:52:24.122458 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e925686f85164a9b94c4c280ebebfadb" } modify_external_catalogs: true
I20250625 07:52:24.122597 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e925686f85164a9b94c4c280ebebfadb" } modify_external_catalogs: true
W20250625 07:52:24.123685 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bfb421ea711a461890e267153250acea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb421ea711a461890e267153250acea already in progress: creating tablet
W20250625 07:52:24.124030 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a40dacaa394747aaa100a0527e14eed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40dacaa394747aaa100a0527e14eed0 already in progress: creating tablet
W20250625 07:52:24.124378 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9f655669b74b4a6594fbd63c45d15bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f655669b74b4a6594fbd63c45d15bf4 already in progress: creating tablet
I20250625 07:52:24.125912 744 tablet_service.cc:1468] Processing CreateTablet for tablet d605113ad8f246148825923e5bcdf00d (DEFAULT_TABLE table=default.Table_ca8f9a686ffb42c18e83701403816a41 [id=a433772a26974cd8baafdf1ae952401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.125962 884 tablet_service.cc:1468] Processing CreateTablet for tablet d605113ad8f246148825923e5bcdf00d (DEFAULT_TABLE table=default.Table_ca8f9a686ffb42c18e83701403816a41 [id=a433772a26974cd8baafdf1ae952401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.126308 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d605113ad8f246148825923e5bcdf00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.126344 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d605113ad8f246148825923e5bcdf00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.129961 750 tablet_service.cc:1468] Processing CreateTablet for tablet 57f763f08289455995c930d9ee03c95c (DEFAULT_TABLE table=default.Table_de8edff1052842d9bd91dc30913c757b [id=b647843c9ebc4dfc81e47a1856dee805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.130366 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f763f08289455995c930d9ee03c95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.133008 749 tablet_service.cc:1468] Processing CreateTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 (DEFAULT_TABLE table=default.Table_3d40c073684a48a69c3c60f46aab48bc [id=cd0d600c29a6482c9b3456d98d6c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.133388 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a30941e3b9c475db17f961c38d96ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.133863 884 tablet_service.cc:1468] Processing CreateTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 (DEFAULT_TABLE table=default.Table_31f10020a22b49bc8778c19390b27fd0 [id=823be989d0474a8393696bf529a988d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.134200 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2ced3d2fba4f74a5d89d4bc5580905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.136849 893 tablet_service.cc:1468] Processing CreateTablet for tablet 57f763f08289455995c930d9ee03c95c (DEFAULT_TABLE table=default.Table_de8edff1052842d9bd91dc30913c757b [id=b647843c9ebc4dfc81e47a1856dee805]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.136911 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 (DEFAULT_TABLE table=default.Table_31f10020a22b49bc8778c19390b27fd0 [id=823be989d0474a8393696bf529a988d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.137203 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f763f08289455995c930d9ee03c95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.137333 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2ced3d2fba4f74a5d89d4bc5580905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.138729 1160 ts_tablet_manager.cc:1918] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.138837 1160 log.cc:1199] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:24.139225 1160 ts_tablet_manager.cc:1939] T 2db9e8d0fefa47bfaeee462dba096b49 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.142345 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 184cd64146f141db8e870af121a7772f
I20250625 07:52:24.142509 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34] alter complete (version 3)
I20250625 07:52:24.142853 594 catalog_manager.cc:6652] ReplaceTablet: tablet 184cd64146f141db8e870af121a7772f deleted and replaced by tablet 28263d8e0fee4520b16704a57ab9f4b4
I20250625 07:52:24.143522 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 84f47caf3bae41b79b0aa1e320f15da4
I20250625 07:52:24.143751 755 tablet_service.cc:1468] Processing CreateTablet for tablet 81d13afd21f34e8794b603ccaa0b1d84 (DEFAULT_TABLE table=default.Table_34ac7b0478f94f3d8a0c39d6671c95e8 [id=8ad1b460c1314a86a0d2de16d1f15f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.143966 596 catalog_manager.cc:6652] ReplaceTablet: tablet 84f47caf3bae41b79b0aa1e320f15da4 deleted and replaced by tablet cac7d20327e14f90af25acb58d724b68
I20250625 07:52:24.144196 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d13afd21f34e8794b603ccaa0b1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.148665 1160 tablet_replica.cc:331] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.150162 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2db9e8d0fefa47bfaeee462dba096b49 (table default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]) successfully deleted
W20250625 07:52:24.150558 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 11083173ebae45bc8c567cf5b750a06f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11083173ebae45bc8c567cf5b750a06f already in progress: creating tablet
W20250625 07:52:24.150851 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7db12504a1a54add920561dae27d2d2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db12504a1a54add920561dae27d2d2b already in progress: creating tablet
W20250625 07:52:24.151152 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 91a1d44dde9942a2bfd32a108b9dea92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91a1d44dde9942a2bfd32a108b9dea92 already in progress: creating tablet
W20250625 07:52:24.151466 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6e77ff1b8ca848ff8e1b4538645b4be2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e77ff1b8ca848ff8e1b4538645b4be2 already in progress: creating tablet
W20250625 07:52:24.151748 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 90f2bef7baa54f1a96b3b59921d59457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90f2bef7baa54f1a96b3b59921d59457 already in progress: creating tablet
I20250625 07:52:24.151791 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:24.152019 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 065822c50586488785f7d9d53841a4dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 065822c50586488785f7d9d53841a4dd already in progress: creating tablet
I20250625 07:52:24.152069 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:24.152210 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bfb421ea711a461890e267153250acea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb421ea711a461890e267153250acea already in progress: creating tablet
I20250625 07:52:24.152240 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 184cd64146f141db8e870af121a7772f with delete_type TABLET_DATA_DELETED (replaced by tablet 28263d8e0fee4520b16704a57ab9f4b4) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.152374 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:24.152395 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a40dacaa394747aaa100a0527e14eed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40dacaa394747aaa100a0527e14eed0 already in progress: creating tablet
W20250625 07:52:24.152573 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9f655669b74b4a6594fbd63c45d15bf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f655669b74b4a6594fbd63c45d15bf4 already in progress: creating tablet
W20250625 07:52:24.152803 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9bccda893fd5466886a4d253172309e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bccda893fd5466886a4d253172309e1 already in progress: creating tablet
I20250625 07:52:24.152667 754 tablet_service.cc:1468] Processing CreateTablet for tablet bd9a6cded85c4d6296f8468644de2016 (DEFAULT_TABLE table=default.Table_cf47515a30a347929633a53b20a19b4b [id=d0a78a1372fa4f7693f95f4ed2dd6f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.148984 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.149437 889 tablet_service.cc:1468] Processing CreateTablet for tablet 81d13afd21f34e8794b603ccaa0b1d84 (DEFAULT_TABLE table=default.Table_34ac7b0478f94f3d8a0c39d6671c95e8 [id=8ad1b460c1314a86a0d2de16d1f15f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.153110 1160 raft_consensus.cc:2241] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.153200 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a6cded85c4d6296f8468644de2016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.149562 891 tablet_service.cc:1468] Processing CreateTablet for tablet bd9a6cded85c4d6296f8468644de2016 (DEFAULT_TABLE table=default.Table_cf47515a30a347929633a53b20a19b4b [id=d0a78a1372fa4f7693f95f4ed2dd6f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.153280 1160 raft_consensus.cc:2270] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.153296 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d13afd21f34e8794b603ccaa0b1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.153817 1160 ts_tablet_manager.cc:1905] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.155057 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:24.152806 742 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.149770 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.149857 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 184cd64146f141db8e870af121a7772f with delete_type TABLET_DATA_DELETED (replaced by tablet 28263d8e0fee4520b16704a57ab9f4b4) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.156869 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a6cded85c4d6296f8468644de2016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.164151 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:24.164559 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:24.166975 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 267f9accd9b34154aa0896055a930c73 (DEFAULT_TABLE table=default.Table_ba73f3114ea04e219b7a53b12922b3ea [id=fdd3d4b230684386a010ad62e0a2bc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.169474 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.169780 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.172024 1166 ts_tablet_manager.cc:1918] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.172277 1166 log.cc:1199] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0024618293d74f559b04258f66616044
I20250625 07:52:24.174186 1166 ts_tablet_manager.cc:1939] T 0024618293d74f559b04258f66616044 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.174885 1160 ts_tablet_manager.cc:1918] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.174999 1160 log.cc:1199] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/29f2997d1d714100950383e6af04f315
I20250625 07:52:24.175288 1160 ts_tablet_manager.cc:1939] T 29f2997d1d714100950383e6af04f315 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.176340 1163 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.176476 1163 ts_tablet_manager.cc:1397] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.005s
I20250625 07:52:24.179580 1163 raft_consensus.cc:357] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.179761 1163 raft_consensus.cc:383] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.179821 1163 raft_consensus.cc:738] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.179812 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 29f2997d1d714100950383e6af04f315 (table default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]) successfully deleted
I20250625 07:52:24.180024 1160 tablet_replica.cc:331] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.180007 1163 consensus_queue.cc:260] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.180501 1163 ts_tablet_manager.cc:1428] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250625 07:52:24.180613 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 07a07af928ff4d3d894b074ef0cf44eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a07af928ff4d3d894b074ef0cf44eb already in progress: creating tablet
I20250625 07:52:24.180806 1163 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:24.180889 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fcc3afc210a540058294121d2b83c66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcc3afc210a540058294121d2b83c66d already in progress: creating tablet
W20250625 07:52:24.181097 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f6fbe5526772435eb6ff141715994b65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fbe5526772435eb6ff141715994b65 already in progress: creating tablet
I20250625 07:52:24.182356 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267f9accd9b34154aa0896055a930c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.185441 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ef471c579fc142c8a111b91165d97c5c
I20250625 07:52:24.186986 743 tablet_service.cc:1515] Processing DeleteTablet for tablet ef471c579fc142c8a111b91165d97c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.188458 589 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 28263d8e0fee4520b16704a57ab9f4b4
I20250625 07:52:24.188647 1617 raft_consensus.cc:491] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.188827 1617 raft_consensus.cc:513] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.189191 1543 raft_consensus.cc:491] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.189384 1543 raft_consensus.cc:513] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.189594 1617 leader_election.cc:290] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.190958 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:24.191071 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ef471c579fc142c8a111b91165d97c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.191165 912 raft_consensus.cc:2466] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:24.191255 1163 tablet_bootstrap.cc:654] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.191493 1543 leader_election.cc:290] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.191977 713 leader_election.cc:304] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:24.192404 1543 raft_consensus.cc:2802] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:24.192575 1543 raft_consensus.cc:491] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:24.192788 1543 raft_consensus.cc:3058] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:24.194530 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:24.194787 915 raft_consensus.cc:2466] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:24.194988 1162 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.195211 1162 ts_tablet_manager.cc:1397] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.001s
I20250625 07:52:24.195369 713 leader_election.cc:304] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:24.195708 1617 raft_consensus.cc:2802] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:24.195825 1617 raft_consensus.cc:491] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:24.195905 1617 raft_consensus.cc:3058] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:24.196107 1162 raft_consensus.cc:357] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.196385 1162 raft_consensus.cc:383] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.196519 1162 raft_consensus.cc:738] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.197036 1162 consensus_queue.cc:260] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.197983 1162 ts_tablet_manager.cc:1428] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:24.200908 1162 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.200908 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 (DEFAULT_TABLE table=default.Table_3d40c073684a48a69c3c60f46aab48bc [id=cd0d600c29a6482c9b3456d98d6c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.202911 1162 tablet_bootstrap.cc:654] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.222827 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e67edafbb8384a69ab2a92fd5fabd83f" } new_table_name: "default.Table_2aa50fc1fa7542bb8d872216447d73f9" modify_external_catalogs: true
I20250625 07:52:24.224375 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_377ef3ec98894b328b266b665f9a75f0" } new_table_name: "default.Table_891a8459d2ca484ba38e14d7f5768fdb" modify_external_catalogs: true
I20250625 07:52:24.227607 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.228078 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:24.228657 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 07a07af928ff4d3d894b074ef0cf44eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a07af928ff4d3d894b074ef0cf44eb already in progress: creating tablet
W20250625 07:52:24.229110 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fcc3afc210a540058294121d2b83c66d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcc3afc210a540058294121d2b83c66d already in progress: creating tablet
I20250625 07:52:24.229398 1166 tablet_replica.cc:331] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.229657 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ac08bcc7329545ba8ba93e26d1c810f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.229804 1166 raft_consensus.cc:2241] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.229982 1166 raft_consensus.cc:2270] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.229991 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0024618293d74f559b04258f66616044 (table default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]) successfully deleted
W20250625 07:52:24.230252 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac08bcc7329545ba8ba93e26d1c810f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:24.230296 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 84f47caf3bae41b79b0aa1e320f15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f47caf3bae41b79b0aa1e320f15da4 already in progress: creating tablet
I20250625 07:52:24.230715 1166 ts_tablet_manager.cc:1905] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.231527 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_83a512e0c6d648cc857a484855ea1264"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.232043 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83a512e0c6d648cc857a484855ea1264 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:24.232463 713 consensus_peers.cc:487] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:24.232853 713 consensus_peers.cc:487] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:24.233323 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b2cb8c1f912f43bfad73814acbecbc34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.233625 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b160757deef94a008c500d961bbb19ab has no consensus state
W20250625 07:52:24.233824 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2cb8c1f912f43bfad73814acbecbc34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:24.233824 713 leader_election.cc:343] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:24.234270 713 leader_election.cc:343] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.234963 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3525675ad75a431abb04ceb227487582" } modify_external_catalogs: true
I20250625 07:52:24.234982 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet d2c88638ac1843649fa7060cf1c62f90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.235127 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3525675ad75a431abb04ceb227487582" } modify_external_catalogs: true
I20250625 07:52:24.235416 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet f6fbe5526772435eb6ff141715994b65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.235888 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 875fa1b46903452eaed1f2235d63dfdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.236565 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 7d42f5f3910742f0942742eb77ea1435 (DEFAULT_TABLE table=default.Table_58ba6295501d43f4b2c150eb6ac31aa2 [id=e78e1f8faa4c4ac5b908fca952514a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.237178 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d42f5f3910742f0942742eb77ea1435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.239427 1617 raft_consensus.cc:513] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.240162 1617 leader_election.cc:290] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.242869 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8904912af22c440e8d22530ea0c38f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.243533 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8904912af22c440e8d22530ea0c38f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.243549 1543 raft_consensus.cc:513] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.244241 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d605113ad8f246148825923e5bcdf00d (DEFAULT_TABLE table=default.Table_ca8f9a686ffb42c18e83701403816a41 [id=a433772a26974cd8baafdf1ae952401d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.244647 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d605113ad8f246148825923e5bcdf00d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.245925 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 (DEFAULT_TABLE table=default.Table_31f10020a22b49bc8778c19390b27fd0 [id=823be989d0474a8393696bf529a988d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.246093 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 (DEFAULT_TABLE table=default.Table_3d40c073684a48a69c3c60f46aab48bc [id=cd0d600c29a6482c9b3456d98d6c4a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.246188 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 57f763f08289455995c930d9ee03c95c (DEFAULT_TABLE table=default.Table_de8edff1052842d9bd91dc30913c757b [id=b647843c9ebc4dfc81e47a1856dee805]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.246263 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64ca83369ffe46ea9bd756f20da936cf has no consensus state
I20250625 07:52:24.246379 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c2ced3d2fba4f74a5d89d4bc5580905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.246590 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a30941e3b9c475db17f961c38d96ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.246973 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f763f08289455995c930d9ee03c95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.247838 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 81d13afd21f34e8794b603ccaa0b1d84 (DEFAULT_TABLE table=default.Table_34ac7b0478f94f3d8a0c39d6671c95e8 [id=8ad1b460c1314a86a0d2de16d1f15f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.248338 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81d13afd21f34e8794b603ccaa0b1d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.248863 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.252902 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 184cd64146f141db8e870af121a7772f with delete_type TABLET_DATA_DELETED (replaced by tablet 28263d8e0fee4520b16704a57ab9f4b4) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.253371 1013 tablet_service.cc:1468] Processing CreateTablet for tablet bd9a6cded85c4d6296f8468644de2016 (DEFAULT_TABLE table=default.Table_cf47515a30a347929633a53b20a19b4b [id=d0a78a1372fa4f7693f95f4ed2dd6f43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.253767 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a6cded85c4d6296f8468644de2016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.254715 1543 leader_election.cc:290] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.256824 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.260957 1166 ts_tablet_manager.cc:1918] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.261129 1166 log.cc:1199] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:24.261546 1166 ts_tablet_manager.cc:1939] T 2d8d3300befc4f419070ae84725ee96a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.262634 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2d8d3300befc4f419070ae84725ee96a (table default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc [id=f24c46ab73494d2b958f0fa750eadc63]) successfully deleted
I20250625 07:52:24.271806 1543 raft_consensus.cc:491] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.272032 1543 raft_consensus.cc:513] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.243724 1026 tablet_service.cc:1468] Processing CreateTablet for tablet a02593ec05be42989afc7e2487cfb7e0 (DEFAULT_TABLE table=default.Table_4e3a4071789f48b5a7b5ab2c208f5130 [id=ae6dca9d3beb488aa00a15d2307bd0d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.273706 713 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 651c74bf9a9b47d6ab5a875a5c304783
I20250625 07:52:24.273898 1543 leader_election.cc:290] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.274142 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02593ec05be42989afc7e2487cfb7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.274920 1166 tablet_replica.cc:331] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.275225 1166 raft_consensus.cc:2241] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.294203 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d85c53889892469d956fb4d28e74e2b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.294942 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d85c53889892469d956fb4d28e74e2b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.296417 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce3de32cf03042d6990e81e54f461669" } modify_external_catalogs: true
I20250625 07:52:24.296623 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce3de32cf03042d6990e81e54f461669" } modify_external_catalogs: true
I20250625 07:52:24.297317 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2c27be0b9107412facdd1bc4d768eebb" } modify_external_catalogs: true
I20250625 07:52:24.297492 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2c27be0b9107412facdd1bc4d768eebb" } modify_external_catalogs: true
I20250625 07:52:24.298081 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e2bd1f5414044ed5bc39851226d92171" } new_table_name: "default.Table_bf663ee0009a49b6839fc491f911313e" modify_external_catalogs: true
I20250625 07:52:24.300447 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9e0c2aded59d426c9ce157d9d9038958"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.301025 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e0c2aded59d426c9ce157d9d9038958 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.302477 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b252524928e4ed3a800c849b1ffd8e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.303092 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b252524928e4ed3a800c849b1ffd8e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.310442 594 master_service.cc:948] ReplaceTablet: received request to replace tablet 81976db0a2f94128a89842a12f067231 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:24.312026 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7d746365978745ddb6d89f0e71262bad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.312588 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d746365978745ddb6d89f0e71262bad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.203140 1651 raft_consensus.cc:491] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.220715 1705 raft_consensus.cc:491] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.316984 1651 raft_consensus.cc:513] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.317014 1705 raft_consensus.cc:513] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.235235 1729 raft_consensus.cc:491] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.244589 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:24.317289 912 raft_consensus.cc:3058] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:24.317265 1729 raft_consensus.cc:513] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.317655 1651 leader_election.cc:290] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.317734 1729 leader_election.cc:290] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.318558 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.318656 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:24.318979 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.319067 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "960ded22e2eb42d9a3494c020a4ea354" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:24.317665 1705 leader_election.cc:290] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.319406 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:24.319816 850 leader_election.cc:343] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:24.319969 850 leader_election.cc:343] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.320194 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "960ded22e2eb42d9a3494c020a4ea354" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.320585 912 raft_consensus.cc:2466] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:24.320667 850 leader_election.cc:343] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:24.320734 851 leader_election.cc:343] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.320873 851 leader_election.cc:304] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.321053 713 leader_election.cc:304] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:24.321316 1543 raft_consensus.cc:2802] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 07:52:24.321313 851 leader_election.cc:343] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.321413 1543 raft_consensus.cc:695] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:24.321420 851 leader_election.cc:304] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.321673 1729 raft_consensus.cc:2747] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.321632 1543 consensus_queue.cc:237] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:24.321800 851 leader_election.cc:343] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.321812 1729 raft_consensus.cc:2747] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.321885 851 leader_election.cc:304] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.322057 1729 raft_consensus.cc:2747] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.323257 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a30941e3b9c475db17f961c38d96ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.316761 1160 raft_consensus.cc:2241] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.245375 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:24.324779 915 raft_consensus.cc:3058] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:24.324791 1160 raft_consensus.cc:2270] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.326135 1160 ts_tablet_manager.cc:1905] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.328461 915 raft_consensus.cc:2466] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:24.329116 713 leader_election.cc:304] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:24.329406 1617 raft_consensus.cc:2802] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:24.329542 1617 raft_consensus.cc:695] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:24.329775 1617 consensus_queue.cc:237] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.330210 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f98ca9d11df0483887149bfbb92885be" } new_table_name: "default.Table_1bf45c7de2ca4492bbff31a7ecd9404a" modify_external_catalogs: true
I20250625 07:52:24.330469 1158 tablet_bootstrap.cc:492] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:24.330649 1158 ts_tablet_manager.cc:1397] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.490s user 0.001s sys 0.006s
I20250625 07:52:24.330833 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a167c7e87544c0ea7c10b4496629916" } new_table_name: "default.Table_db02a192704e4d678f0532f5c8f9694d" modify_external_catalogs: true
I20250625 07:52:24.331218 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 4 got 0
W20250625 07:52:24.331715 595 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 1228018baf3340d99b3f94b150d0d533 consensus state has no leader
I20250625 07:52:24.335620 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c2765164e15a4bc8adeae4612f437ce2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.336397 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2765164e15a4bc8adeae4612f437ce2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.338639 1158 raft_consensus.cc:357] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:24.338866 1158 raft_consensus.cc:383] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.338948 1158 raft_consensus.cc:738] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.339190 1158 consensus_queue.cc:260] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:24.341181 750 tablet_service.cc:1468] Processing CreateTablet for tablet d705b102654747b5a9f16ae9290f96b4 (DEFAULT_TABLE table=default.Table_f678232d9ed444da9031bda1a4bf561e [id=02f47cef001a4202994fad0596557c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.341396 747 tablet_service.cc:1468] Processing CreateTablet for tablet 6009f781491c4481b5a2acdfdb2b8548 (DEFAULT_TABLE table=default.Table_15c9f1d9e9e842ef8ce4b125b9350af3 [id=4b519d402ab84b07a620ed1176a3611f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.341611 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d705b102654747b5a9f16ae9290f96b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.341765 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6009f781491c4481b5a2acdfdb2b8548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.343084 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:24.343361 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "651c74bf9a9b47d6ab5a875a5c304783" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.343508 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet ef471c579fc142c8a111b91165d97c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.344197 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:24.345232 749 tablet_service.cc:1468] Processing CreateTablet for tablet 71b9f3d6de464a218e08e47c1eec544b (DEFAULT_TABLE table=default.Table_3446d001da73487c8634225bf869b5a4 [id=dc6ade88fcfc40e49e250f9510713956]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.345742 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b9f3d6de464a218e08e47c1eec544b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.349099 754 tablet_service.cc:1468] Processing CreateTablet for tablet 64ca83369ffe46ea9bd756f20da936cf (DEFAULT_TABLE table=default.Table_2aa50fc1fa7542bb8d872216447d73f9 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.349586 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ca83369ffe46ea9bd756f20da936cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.353113 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1844a29f83614b72ad6a230cd800bebb (DEFAULT_TABLE table=default.Table_bcd09e33bddc4315879e476a6897ba21 [id=a0a2e168e2534d6d8875adae8b32f759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.353543 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1844a29f83614b72ad6a230cd800bebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.354274 1163 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.354418 1163 ts_tablet_manager.cc:1397] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.174s user 0.004s sys 0.009s
I20250625 07:52:24.355208 1734 raft_consensus.cc:491] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.355545 1734 raft_consensus.cc:513] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:24.356472 1734 leader_election.cc:290] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:24.357045 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4e345f039ca347b1a70b3eed28e58ceb (DEFAULT_TABLE table=default.Table_0fd3c55764824ab883ec46fb92e6fdee [id=f7c20436cd244d79a7c4af5fd8077296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.357432 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e345f039ca347b1a70b3eed28e58ceb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.358825 986 leader_election.cc:343] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 4384bc544af149338514d40adf683f34
I20250625 07:52:24.359310 1163 raft_consensus.cc:357] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.359577 1163 raft_consensus.cc:383] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.359751 1163 raft_consensus.cc:738] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.360013 1163 consensus_queue.cc:260] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:24.362775 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d2c88638ac1843649fa7060cf1c62f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c88638ac1843649fa7060cf1c62f90 already in progress: creating tablet
W20250625 07:52:24.363102 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f6fbe5526772435eb6ff141715994b65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6fbe5526772435eb6ff141715994b65 already in progress: creating tablet
W20250625 07:52:24.363317 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 875fa1b46903452eaed1f2235d63dfdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 875fa1b46903452eaed1f2235d63dfdf already in progress: creating tablet
W20250625 07:52:24.363618 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 84f47caf3bae41b79b0aa1e320f15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f47caf3bae41b79b0aa1e320f15da4 already in progress: creating tablet
I20250625 07:52:24.364346 1163 ts_tablet_manager.cc:1428] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 07:52:24.365777 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6009f781491c4481b5a2acdfdb2b8548 (DEFAULT_TABLE table=default.Table_15c9f1d9e9e842ef8ce4b125b9350af3 [id=4b519d402ab84b07a620ed1176a3611f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.366185 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6009f781491c4481b5a2acdfdb2b8548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.366112 881 tablet_service.cc:1468] Processing CreateTablet for tablet d705b102654747b5a9f16ae9290f96b4 (DEFAULT_TABLE table=default.Table_f678232d9ed444da9031bda1a4bf561e [id=02f47cef001a4202994fad0596557c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.366240 889 tablet_service.cc:1468] Processing CreateTablet for tablet 64ca83369ffe46ea9bd756f20da936cf (DEFAULT_TABLE table=default.Table_2aa50fc1fa7542bb8d872216447d73f9 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.366425 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d705b102654747b5a9f16ae9290f96b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.366601 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ca83369ffe46ea9bd756f20da936cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.368232 1163 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.368964 891 tablet_service.cc:1468] Processing CreateTablet for tablet 71b9f3d6de464a218e08e47c1eec544b (DEFAULT_TABLE table=default.Table_3446d001da73487c8634225bf869b5a4 [id=dc6ade88fcfc40e49e250f9510713956]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.369319 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b9f3d6de464a218e08e47c1eec544b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.369330 713 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.369469 713 leader_election.cc:304] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.369724 1660 raft_consensus.cc:2747] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.369951 883 tablet_service.cc:1468] Processing CreateTablet for tablet 1844a29f83614b72ad6a230cd800bebb (DEFAULT_TABLE table=default.Table_bcd09e33bddc4315879e476a6897ba21 [id=a0a2e168e2534d6d8875adae8b32f759]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.370194 713 leader_election.cc:343] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.370383 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4e345f039ca347b1a70b3eed28e58ceb (DEFAULT_TABLE table=default.Table_0fd3c55764824ab883ec46fb92e6fdee [id=f7c20436cd244d79a7c4af5fd8077296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.374609 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e345f039ca347b1a70b3eed28e58ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.375242 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1844a29f83614b72ad6a230cd800bebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.386129 1158 ts_tablet_manager.cc:1428] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.055s user 0.001s sys 0.002s
I20250625 07:52:24.386837 1158 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:24.387135 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 1844a29f83614b72ad6a230cd800bebb (DEFAULT_TABLE table=default.Table_bcd09e33bddc4315879e476a6897ba21 [id=a0a2e168e2534d6d8875adae8b32f759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.387667 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1844a29f83614b72ad6a230cd800bebb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.388399 713 leader_election.cc:343] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.388746 1158 tablet_bootstrap.cc:654] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.392342 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d705b102654747b5a9f16ae9290f96b4 (DEFAULT_TABLE table=default.Table_f678232d9ed444da9031bda1a4bf561e [id=02f47cef001a4202994fad0596557c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.392856 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d705b102654747b5a9f16ae9290f96b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.394492 1160 ts_tablet_manager.cc:1918] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.394614 1160 log.cc:1199] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/875fa1b46903452eaed1f2235d63dfdf
I20250625 07:52:24.396090 1162 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.396242 1162 ts_tablet_manager.cc:1397] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.195s user 0.061s sys 0.045s
I20250625 07:52:24.396638 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 71b9f3d6de464a218e08e47c1eec544b (DEFAULT_TABLE table=default.Table_3446d001da73487c8634225bf869b5a4 [id=dc6ade88fcfc40e49e250f9510713956]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.397275 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b9f3d6de464a218e08e47c1eec544b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.397223 1162 raft_consensus.cc:357] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.397415 1162 raft_consensus.cc:383] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.397471 1162 raft_consensus.cc:738] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.397660 1162 consensus_queue.cc:260] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.399410 1162 ts_tablet_manager.cc:1428] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:24.401181 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4e345f039ca347b1a70b3eed28e58ceb (DEFAULT_TABLE table=default.Table_0fd3c55764824ab883ec46fb92e6fdee [id=f7c20436cd244d79a7c4af5fd8077296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.401801 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e345f039ca347b1a70b3eed28e58ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.405107 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6009f781491c4481b5a2acdfdb2b8548 (DEFAULT_TABLE table=default.Table_15c9f1d9e9e842ef8ce4b125b9350af3 [id=4b519d402ab84b07a620ed1176a3611f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.405725 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6009f781491c4481b5a2acdfdb2b8548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.409174 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b160757deef94a008c500d961bbb19ab (DEFAULT_TABLE table=default.Table_891a8459d2ca484ba38e14d7f5768fdb [id=38efffc9521b4c73a1ba691fa0e522cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.409795 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b160757deef94a008c500d961bbb19ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.409809 1162 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.411751 1162 tablet_bootstrap.cc:654] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.416568 1166 raft_consensus.cc:2270] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:24.425215 985 leader_election.cc:343] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4384bc544af149338514d40adf683f34
I20250625 07:52:24.425513 985 leader_election.cc:304] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:24.425844 1733 raft_consensus.cc:2747] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.426869 1705 raft_consensus.cc:491] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.427028 1705 raft_consensus.cc:513] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.427508 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 64ca83369ffe46ea9bd756f20da936cf (DEFAULT_TABLE table=default.Table_2aa50fc1fa7542bb8d872216447d73f9 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.427654 1705 leader_election.cc:290] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.427947 1163 tablet_bootstrap.cc:654] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.427999 1160 ts_tablet_manager.cc:1939] T 875fa1b46903452eaed1f2235d63dfdf P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.428433 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:24.429003 851 leader_election.cc:343] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.429093 1160 tablet_replica.cc:331] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.429276 1160 raft_consensus.cc:2241] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.429420 1160 raft_consensus.cc:2270] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.429455 1166 ts_tablet_manager.cc:1905] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.429736 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.430100 1160 ts_tablet_manager.cc:1905] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:24.430212 850 leader_election.cc:343] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.430331 850 leader_election.cc:304] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.430670 1705 raft_consensus.cc:2747] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.436084 1160 ts_tablet_manager.cc:1918] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.436084 1166 ts_tablet_manager.cc:1918] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.436225 1160 log.cc:1199] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:24.436242 1166 log.cc:1199] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:24.436543 1160 ts_tablet_manager.cc:1939] T 960ded22e2eb42d9a3494c020a4ea354 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.436548 1166 ts_tablet_manager.cc:1939] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.439961 1166 tablet_replica.cc:331] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.440074 1163 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.440202 1163 ts_tablet_manager.cc:1397] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.000s
I20250625 07:52:24.440336 1166 raft_consensus.cc:2241] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.440551 1166 raft_consensus.cc:2270] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.440985 1163 raft_consensus.cc:357] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.441157 1163 raft_consensus.cc:383] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.441218 1163 raft_consensus.cc:738] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.441426 1163 consensus_queue.cc:260] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.441654 1166 ts_tablet_manager.cc:1905] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.443850 1163 ts_tablet_manager.cc:1428] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:24.444154 1163 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.448338 1163 tablet_bootstrap.cc:654] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.453505 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 4 got 0
I20250625 07:52:24.454188 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 250175f54ba94a9fbc2455d9cf27fb2e from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:24.454388 1166 ts_tablet_manager.cc:1918] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.454486 1166 log.cc:1199] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b5f917cfd2384a72920847f8b89358b9
I20250625 07:52:24.454780 1166 ts_tablet_manager.cc:1939] T b5f917cfd2384a72920847f8b89358b9 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.455806 1166 tablet_replica.cc:331] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.456066 1166 raft_consensus.cc:2241] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.456216 1163 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.456264 1166 raft_consensus.cc:2270] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.456379 1163 ts_tablet_manager.cc:1397] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
W20250625 07:52:24.456723 584 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 50738.5us
I20250625 07:52:24.457175 1166 ts_tablet_manager.cc:1905] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.457212 1163 raft_consensus.cc:357] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.457489 1163 raft_consensus.cc:383] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.457559 1163 raft_consensus.cc:738] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.457906 1163 consensus_queue.cc:260] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.458566 754 tablet_service.cc:1468] Processing CreateTablet for tablet b160757deef94a008c500d961bbb19ab (DEFAULT_TABLE table=default.Table_891a8459d2ca484ba38e14d7f5768fdb [id=38efffc9521b4c73a1ba691fa0e522cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.458926 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b160757deef94a008c500d961bbb19ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.459113 744 tablet_service.cc:1468] Processing CreateTablet for tablet 95d6563421c349c59697d1b9f197d54d (DEFAULT_TABLE table=default.Table_ca3d0678d40245259f27ff872ef5ac3d [id=152fb0d4f8f4454e975e26105f59d84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.459457 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d6563421c349c59697d1b9f197d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.460983 749 tablet_service.cc:1468] Processing CreateTablet for tablet 61ae72c767d14c3c8102b6be8a039690 (DEFAULT_TABLE table=default.Table_7a175cc57ae741328a3d515859d7a98a [id=eb72c6161ef44da3a55baa4749305acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.461323 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae72c767d14c3c8102b6be8a039690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.461431 1163 ts_tablet_manager.cc:1428] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:24.461724 1163 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.464630 1163 tablet_bootstrap.cc:654] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.464993 747 tablet_service.cc:1468] Processing CreateTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f (DEFAULT_TABLE table=default.Table_212082b707ae4f5689a2dcffa943a19c [id=f5510bc22132483cbad277ec1f858eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.465341 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbf1cfac4e14a20b3fb53607cdf100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.465766 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (table default.Table_12188272bb8e4c26bb94db8721d37c26 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
I20250625 07:52:24.466044 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b5f917cfd2384a72920847f8b89358b9 (table default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]) successfully deleted
W20250625 07:52:24.466423 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28dddfa84bc1497a962c718dad7d3b6b consensus state has no leader
I20250625 07:52:24.466778 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 08bb47266e8b4a0fa5ec613560b8549e from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:24.466971 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ef471c579fc142c8a111b91165d97c5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef471c579fc142c8a111b91165d97c5c already in progress: creating tablet
W20250625 07:52:24.467090 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 101362us
I20250625 07:52:24.471582 885 tablet_service.cc:1468] Processing CreateTablet for tablet 61ae72c767d14c3c8102b6be8a039690 (DEFAULT_TABLE table=default.Table_7a175cc57ae741328a3d515859d7a98a [id=eb72c6161ef44da3a55baa4749305acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.471984 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae72c767d14c3c8102b6be8a039690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.472214 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]) successfully deleted
W20250625 07:52:24.472574 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d2c88638ac1843649fa7060cf1c62f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2c88638ac1843649fa7060cf1c62f90 already in progress: creating tablet
I20250625 07:52:24.474097 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 960ded22e2eb42d9a3494c020a4ea354 (table default.Table_786d6b63b6904ffe87ce1f3eb9769954 [id=e8fb18d5b2ae454aa4f4a65fab67c467]) successfully deleted
W20250625 07:52:24.474334 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 84f47caf3bae41b79b0aa1e320f15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f47caf3bae41b79b0aa1e320f15da4 already in progress: creating tablet
W20250625 07:52:24.474720 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ef471c579fc142c8a111b91165d97c5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef471c579fc142c8a111b91165d97c5c already in progress: creating tablet
I20250625 07:52:24.475472 883 tablet_service.cc:1468] Processing CreateTablet for tablet 28263d8e0fee4520b16704a57ab9f4b4 (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.475849 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28263d8e0fee4520b16704a57ab9f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.477147 589 catalog_manager.cc:5582] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:24.477839 589 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_da75eea0e9054b62ac5c228ce0fa5b27 [id=b0f32fb69f33447db7620299affa88c8]). Expected version 2 got 0
I20250625 07:52:24.478904 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_06029eba0bd241169e60166d8be6654b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.479058 750 tablet_service.cc:1468] Processing CreateTablet for tablet f64c42861e60451e92131a435f95b012 (DEFAULT_TABLE table=default.Table_497900c260fb4dbaa34bbeb002b540c9 [id=aef3da5c699242dbaeb4791280d2f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.479442 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06029eba0bd241169e60166d8be6654b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.479471 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64c42861e60451e92131a435f95b012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.480603 1660 raft_consensus.cc:491] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.480751 1660 raft_consensus.cc:513] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.481041 889 tablet_service.cc:1468] Processing CreateTablet for tablet 95d6563421c349c59697d1b9f197d54d (DEFAULT_TABLE table=default.Table_ca3d0678d40245259f27ff872ef5ac3d [id=152fb0d4f8f4454e975e26105f59d84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.481388 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d6563421c349c59697d1b9f197d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.481371 1660 leader_election.cc:290] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:24.483337 713 leader_election.cc:343] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:24.486685 743 tablet_service.cc:1468] Processing CreateTablet for tablet cac7d20327e14f90af25acb58d724b68 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.487079 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7d20327e14f90af25acb58d724b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.489178 881 tablet_service.cc:1468] Processing CreateTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f (DEFAULT_TABLE table=default.Table_212082b707ae4f5689a2dcffa943a19c [id=f5510bc22132483cbad277ec1f858eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.489459 1163 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.489537 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbf1cfac4e14a20b3fb53607cdf100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.489583 1163 ts_tablet_manager.cc:1397] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250625 07:52:24.490365 1163 raft_consensus.cc:357] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.490511 1163 raft_consensus.cc:383] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.490554 1163 raft_consensus.cc:738] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.490595 742 tablet_service.cc:1468] Processing CreateTablet for tablet f2fb351cfc94459996a099ea6618468b (DEFAULT_TABLE table=default.Table_4a167c7e87544c0ea7c10b4496629916 [id=3e4cb2507f7d431e9d0854ce07b09b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.490727 1163 consensus_queue.cc:260] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.490944 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb351cfc94459996a099ea6618468b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.491791 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3670cadfa2841949705c65c9b6c0927" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.492370 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 28263d8e0fee4520b16704a57ab9f4b4 (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.493180 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 95d6563421c349c59697d1b9f197d54d (DEFAULT_TABLE table=default.Table_ca3d0678d40245259f27ff872ef5ac3d [id=152fb0d4f8f4454e975e26105f59d84c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.493350 891 tablet_service.cc:1468] Processing CreateTablet for tablet f64c42861e60451e92131a435f95b012 (DEFAULT_TABLE table=default.Table_497900c260fb4dbaa34bbeb002b540c9 [id=aef3da5c699242dbaeb4791280d2f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.493535 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f (DEFAULT_TABLE table=default.Table_212082b707ae4f5689a2dcffa943a19c [id=f5510bc22132483cbad277ec1f858eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.493767 1163 ts_tablet_manager.cc:1428] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:24.494212 893 tablet_service.cc:1468] Processing CreateTablet for tablet f2fb351cfc94459996a099ea6618468b (DEFAULT_TABLE table=default.Table_4a167c7e87544c0ea7c10b4496629916 [id=3e4cb2507f7d431e9d0854ce07b09b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.494313 1011 tablet_service.cc:1468] Processing CreateTablet for tablet f64c42861e60451e92131a435f95b012 (DEFAULT_TABLE table=default.Table_497900c260fb4dbaa34bbeb002b540c9 [id=aef3da5c699242dbaeb4791280d2f4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.494026 884 tablet_service.cc:1468] Processing CreateTablet for tablet cac7d20327e14f90af25acb58d724b68 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.494490 755 tablet_service.cc:1468] Processing CreateTablet for tablet 28263d8e0fee4520b16704a57ab9f4b4 (DEFAULT_TABLE table=default.Table_2d792b2565234096a0bc44a9c82abc1c [id=68185ee7591e477381ec821d8fb08b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.494537 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cac7d20327e14f90af25acb58d724b68 (DEFAULT_TABLE table=default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.494894 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28263d8e0fee4520b16704a57ab9f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.494936 1543 consensus_queue.cc:1035] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:24.494020 1163 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.495355 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f2fb351cfc94459996a099ea6618468b (DEFAULT_TABLE table=default.Table_4a167c7e87544c0ea7c10b4496629916 [id=3e4cb2507f7d431e9d0854ce07b09b2a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:24.496516 713 leader_election.cc:343] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.496651 713 leader_election.cc:304] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:24.497377 713 consensus_peers.cc:487] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:24.497547 1543 raft_consensus.cc:2747] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.505980 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64c42861e60451e92131a435f95b012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.506559 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb351cfc94459996a099ea6618468b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.507264 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7d20327e14f90af25acb58d724b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.508424 1163 tablet_bootstrap.cc:654] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:24.513906 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2fb351cfc94459996a099ea6618468b consensus state has no leader
I20250625 07:52:24.515684 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ee0a0386d5404189ac77d4d731677aba" } modify_external_catalogs: true
I20250625 07:52:24.515874 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ee0a0386d5404189ac77d4d731677aba" } modify_external_catalogs: true
W20250625 07:52:24.516947 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0352225d645c408ba8c74273ee5f56e6 consensus state has no leader
I20250625 07:52:24.517261 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8cc8ba5db88541abb0f69c1df86ac4be" } new_table_name: "default.Table_0421905c66ea4425b44ec753849b7d6e" modify_external_catalogs: true
I20250625 07:52:24.518460 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 019959e9abed46fbbbb4755cb01cef44
I20250625 07:52:24.519258 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.519191 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d9f68c7739ec412393bfcc8b1440d7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.519754 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f68c7739ec412393bfcc8b1440d7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.520897 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:24.521493 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_091524b607f045b985a03d229bc7c8e4" } new_table_name: "default.Table_aac4a06e92064896958276d8b9438ec0" modify_external_catalogs: true
I20250625 07:52:24.522609 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:24.523020 1705 raft_consensus.cc:491] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.523161 1705 raft_consensus.cc:513] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.523806 1705 leader_election.cc:290] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.524433 895 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.524682 1160 tablet_replica.cc:331] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.524763 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.524888 1160 raft_consensus.cc:2241] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.525038 1160 raft_consensus.cc:2270] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.524763 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.525714 1160 ts_tablet_manager.cc:1905] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.525818 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.525826 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ca0144e3c5142ac9fcbd927b7585a3c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:24.526028 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ba24bbd0c4694930978a7750f07601d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.526036 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ca0144e3c5142ac9fcbd927b7585a3c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:24.526577 851 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.531822 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 08bb47266e8b4a0fa5ec613560b8549e
I20250625 07:52:24.532294 597 catalog_manager.cc:6652] ReplaceTablet: tablet 08bb47266e8b4a0fa5ec613560b8549e deleted and replaced by tablet a25f89bffcd0430a976113d512c6a699
W20250625 07:52:24.534087 850 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.534235 850 leader_election.cc:304] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.534519 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0d7d47bc82394837aff28ef6e5d70c8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.535306 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7d47bc82394837aff28ef6e5d70c8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.535784 1705 raft_consensus.cc:2747] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.536496 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09cf1afe39e740689d3cef2c20f65f84" } modify_external_catalogs: true
I20250625 07:52:24.536655 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09cf1afe39e740689d3cef2c20f65f84" } modify_external_catalogs: true
I20250625 07:52:24.538996 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a4a61d6fd2514d88989f611b35c12baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.539530 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4a61d6fd2514d88989f611b35c12baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.540659 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7a254ead6d2e4a3e9ede91b319970b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.541133 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a254ead6d2e4a3e9ede91b319970b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.543218 881 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.544464 597 catalog_manager.cc:5582] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:24.544965 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:24.547328 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e with delete_type TABLET_DATA_DELETED (replaced by tablet a25f89bffcd0430a976113d512c6a699) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.519764 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dc8c4809369446dc9c12a878a16e6c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.548348 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc8c4809369446dc9c12a878a16e6c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.549906 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e with delete_type TABLET_DATA_DELETED (replaced by tablet a25f89bffcd0430a976113d512c6a699) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.550437 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:24.550412 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f2fb351cfc94459996a099ea6618468b (table default.Table_db02a192704e4d678f0532f5c8f9694d [id=3e4cb2507f7d431e9d0854ce07b09b2a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2fb351cfc94459996a099ea6618468b consensus state has no leader
I20250625 07:52:24.552703 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ba24bbd0c4694930978a7750f07601d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.558183 1160 ts_tablet_manager.cc:1918] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.558385 1160 log.cc:1199] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:24.558969 1160 ts_tablet_manager.cc:1939] T a33b7cab19244ec08d3821b6cc4c072d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.562404 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a33b7cab19244ec08d3821b6cc4c072d (table default.Table_2f89a55bc2694564b94bb87f42761d2a [id=dff8e8b2f6da4c39b78f536bcc8467e5]) successfully deleted
I20250625 07:52:24.562729 1160 tablet_replica.cc:331] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.563184 1160 raft_consensus.cc:2241] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.563512 1160 raft_consensus.cc:2270] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.565130 1160 ts_tablet_manager.cc:1905] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:24.567623 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 019959e9abed46fbbbb4755cb01cef44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019959e9abed46fbbbb4755cb01cef44 already in progress: creating tablet
I20250625 07:52:24.578758 1166 ts_tablet_manager.cc:1918] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.584028 1163 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.585003 1163 ts_tablet_manager.cc:1397] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.000s
I20250625 07:52:24.593676 1166 log.cc:1199] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:24.594311 1166 ts_tablet_manager.cc:1939] T 7ad78944352e4274acd8d9889b1d32b8 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.593902 1163 raft_consensus.cc:357] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.595335 1163 raft_consensus.cc:383] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.596171 1163 raft_consensus.cc:738] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.597966 1166 tablet_replica.cc:331] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.598186 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7ad78944352e4274acd8d9889b1d32b8 (table default.Table_bb291540701c4a4fbb5f3fa31fc4b684 [id=d24d9520ff924056a204dfd910f4dbef]) successfully deleted
I20250625 07:52:24.599056 1163 consensus_queue.cc:260] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.600250 1166 raft_consensus.cc:2241] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.600425 1166 raft_consensus.cc:2270] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.603876 1166 ts_tablet_manager.cc:1905] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.604888 1163 ts_tablet_manager.cc:1428] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250625 07:52:24.607758 1163 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.608883 1160 ts_tablet_manager.cc:1918] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.609736 1160 log.cc:1199] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:24.612524 1160 ts_tablet_manager.cc:1939] T 3ca0144e3c5142ac9fcbd927b7585a3c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.617503 1166 ts_tablet_manager.cc:1918] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.617945 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c1fc4392f58a4d91973506cb0939bb4c
I20250625 07:52:24.618772 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2e1e160bec8647f2b41862f84d3e4817"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.619160 1166 log.cc:1199] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/651c74bf9a9b47d6ab5a875a5c304783
W20250625 07:52:24.619392 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e1e160bec8647f2b41862f84d3e4817 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.619499 1166 ts_tablet_manager.cc:1939] T 651c74bf9a9b47d6ab5a875a5c304783 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.620663 1166 tablet_replica.cc:331] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.621446 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 651c74bf9a9b47d6ab5a875a5c304783 (table default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]) successfully deleted
W20250625 07:52:24.621743 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ef471c579fc142c8a111b91165d97c5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef471c579fc142c8a111b91165d97c5c already in progress: creating tablet
I20250625 07:52:24.621841 1160 tablet_replica.cc:331] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.622665 1166 raft_consensus.cc:2241] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.623046 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c1fc4392f58a4d91973506cb0939bb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.623443 1166 raft_consensus.cc:2270] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:24.623818 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb98ab2850d746a8bf4733efbcc8bdbc consensus state has no leader
I20250625 07:52:24.623454 1160 raft_consensus.cc:2241] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.624878 1166 ts_tablet_manager.cc:1905] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.626080 1163 tablet_bootstrap.cc:654] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.626209 1160 raft_consensus.cc:2270] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.631031 1160 ts_tablet_manager.cc:1905] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.635255 886 tablet_service.cc:1468] Processing CreateTablet for tablet b160757deef94a008c500d961bbb19ab (DEFAULT_TABLE table=default.Table_891a8459d2ca484ba38e14d7f5768fdb [id=38efffc9521b4c73a1ba691fa0e522cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.638868 1166 ts_tablet_manager.cc:1918] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.639555 1166 log.cc:1199] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:24.639191 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 61ae72c767d14c3c8102b6be8a039690 (DEFAULT_TABLE table=default.Table_7a175cc57ae741328a3d515859d7a98a [id=eb72c6161ef44da3a55baa4749305acf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.640015 1166 ts_tablet_manager.cc:1939] T a3670cadfa2841949705c65c9b6c0927 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.641892 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a3670cadfa2841949705c65c9b6c0927 (table default.Table_a881661cd92a4541a73b939c4532acc7 [id=dd2e80e6531a4e3f9be3b2fc19aec035]) successfully deleted
W20250625 07:52:24.642611 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 019959e9abed46fbbbb4755cb01cef44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019959e9abed46fbbbb4755cb01cef44 already in progress: creating tablet
W20250625 07:52:24.642840 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5ec29e4294864fbabdff5a04dcf5b163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec29e4294864fbabdff5a04dcf5b163 already in progress: creating tablet
W20250625 07:52:24.643049 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ba24bbd0c4694930978a7750f07601d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba24bbd0c4694930978a7750f07601d9 already in progress: creating tablet
W20250625 07:52:24.643255 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 08bb47266e8b4a0fa5ec613560b8549e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08bb47266e8b4a0fa5ec613560b8549e already in progress: creating tablet
W20250625 07:52:24.643484 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c1fc4392f58a4d91973506cb0939bb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fc4392f58a4d91973506cb0939bb4c already in progress: creating tablet
I20250625 07:52:24.644963 1158 tablet_bootstrap.cc:492] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:24.645190 1158 ts_tablet_manager.cc:1397] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.258s user 0.025s sys 0.042s
I20250625 07:52:24.646131 1158 raft_consensus.cc:357] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:24.646409 1158 raft_consensus.cc:383] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.646555 1158 raft_consensus.cc:738] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.646831 1158 consensus_queue.cc:260] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:24.647593 1158 ts_tablet_manager.cc:1428] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:24.652264 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.653275 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet ba24bbd0c4694930978a7750f07601d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.654564 1158 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:24.663481 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 4384bc544af149338514d40adf683f34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.665706 1161 tablet_replica.cc:331] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:24.665952 1161 raft_consensus.cc:2241] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.666143 1161 raft_consensus.cc:2270] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.666699 1162 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.666777 1161 ts_tablet_manager.cc:1905] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.667027 1162 ts_tablet_manager.cc:1397] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.257s user 0.025s sys 0.020s
I20250625 07:52:24.669147 1162 raft_consensus.cc:357] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.669457 1162 raft_consensus.cc:383] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.669591 1162 raft_consensus.cc:738] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.669921 1162 consensus_queue.cc:260] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.670075 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ca83369ffe46ea9bd756f20da936cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.670504 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f64c42861e60451e92131a435f95b012. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.670732 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbf1cfac4e14a20b3fb53607cdf100f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.680711 883 tablet_service.cc:1515] Processing DeleteTablet for tablet c1fc4392f58a4d91973506cb0939bb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.681213 1162 ts_tablet_manager.cc:1428] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250625 07:52:24.681648 1162 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.637079 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b160757deef94a008c500d961bbb19ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.684365 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac7d20327e14f90af25acb58d724b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.684749 1162 tablet_bootstrap.cc:654] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.689455 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28263d8e0fee4520b16704a57ab9f4b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.691480 1160 ts_tablet_manager.cc:1918] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.691601 1160 log.cc:1199] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d76681f162074b67a6b886887b5a3d61
I20250625 07:52:24.691913 1160 ts_tablet_manager.cc:1939] T d76681f162074b67a6b886887b5a3d61 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.693223 1161 ts_tablet_manager.cc:1918] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.693548 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d6563421c349c59697d1b9f197d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.696619 1163 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.696789 1163 ts_tablet_manager.cc:1397] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.000s
I20250625 07:52:24.697574 1163 raft_consensus.cc:357] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.697751 1163 raft_consensus.cc:383] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.697819 1163 raft_consensus.cc:738] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.698058 1163 consensus_queue.cc:260] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.698547 1163 ts_tablet_manager.cc:1428] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:24.698866 1163 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.700495 1163 tablet_bootstrap.cc:654] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.701232 1162 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.701406 1162 ts_tablet_manager.cc:1397] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20250625 07:52:24.702490 1162 raft_consensus.cc:357] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.702735 1162 raft_consensus.cc:383] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.702808 1162 raft_consensus.cc:738] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.703095 1162 consensus_queue.cc:260] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.704483 1160 tablet_replica.cc:331] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.704680 1160 raft_consensus.cc:2241] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.704869 1160 raft_consensus.cc:2270] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.705382 1160 ts_tablet_manager.cc:1905] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.706403 1162 ts_tablet_manager.cc:1428] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250625 07:52:24.709101 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fb351cfc94459996a099ea6618468b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.711795 1162 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.713541 1162 tablet_bootstrap.cc:654] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.715878 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3ca0144e3c5142ac9fcbd927b7585a3c (table default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]) successfully deleted
I20250625 07:52:24.716126 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d76681f162074b67a6b886887b5a3d61 (table default.Table_96d96b9fee4d48efaeb2f22ff67b43e6 [id=8340c9a7e5874bf486d4f07192ebb908]) successfully deleted
I20250625 07:52:24.722915 1161 log.cc:1199] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4384bc544af149338514d40adf683f34
I20250625 07:52:24.724104 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.725014 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d6623a19ca9c41db9ea86fb93a800ce4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.725696 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6623a19ca9c41db9ea86fb93a800ce4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.726786 1161 ts_tablet_manager.cc:1939] T 4384bc544af149338514d40adf683f34 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:24.728502 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 387a69222770408598184411225340e7 consensus state has no leader
I20250625 07:52:24.729586 1162 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.729729 1162 ts_tablet_manager.cc:1397] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20250625 07:52:24.730468 1162 raft_consensus.cc:357] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.730667 1162 raft_consensus.cc:383] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.730736 1162 raft_consensus.cc:738] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.730945 1162 consensus_queue.cc:260] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.731451 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c1fc4392f58a4d91973506cb0939bb4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.731705 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_da75eea0e9054b62ac5c228ce0fa5b27" } new_table_name: "default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a" modify_external_catalogs: true
I20250625 07:52:24.733145 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f00b99b9279946bab7c178245d31c414" } modify_external_catalogs: true
I20250625 07:52:24.733314 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f00b99b9279946bab7c178245d31c414" } modify_external_catalogs: true
I20250625 07:52:24.733542 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ae72c767d14c3c8102b6be8a039690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.737018 1162 ts_tablet_manager.cc:1428] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:24.737380 1163 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.737526 1163 ts_tablet_manager.cc:1397] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250625 07:52:24.738286 1163 raft_consensus.cc:357] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.738507 1163 raft_consensus.cc:383] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.738572 1163 raft_consensus.cc:738] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.738765 1163 consensus_queue.cc:260] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.741058 1162 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.742702 1162 tablet_bootstrap.cc:654] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.742779 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e with delete_type TABLET_DATA_DELETED (replaced by tablet a25f89bffcd0430a976113d512c6a699) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.744575 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_31fc7706ba984a4ebeb944ebfb3aeb25" } new_table_name: "default.Table_65740b2f1a624011b8d0248cb3bb723e" modify_external_catalogs: true
I20250625 07:52:24.752758 1163 ts_tablet_manager.cc:1428] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250625 07:52:24.753178 1163 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.755872 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7a798b2471104d69b814df95edc541c6
I20250625 07:52:24.758256 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a798b2471104d69b814df95edc541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:24.759397 1687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:24.759766 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7a798b2471104d69b814df95edc541c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a798b2471104d69b814df95edc541c6 already in progress: creating tablet
W20250625 07:52:24.759984 1687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:24.760154 1687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 07:52:24.762548 883 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.764994 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a798b2471104d69b814df95edc541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:24.765147 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 019959e9abed46fbbbb4755cb01cef44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 019959e9abed46fbbbb4755cb01cef44 already in progress: creating tablet
W20250625 07:52:24.765393 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ba24bbd0c4694930978a7750f07601d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba24bbd0c4694930978a7750f07601d9 already in progress: creating tablet
I20250625 07:52:24.765962 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 4384bc544af149338514d40adf683f34 (table default.Table_ddae42a67be34593a367bf6f3d9e8cde [id=fd174ee9d81e42c19cab3c058584ea20]) successfully deleted
W20250625 07:52:24.766163 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5ec29e4294864fbabdff5a04dcf5b163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec29e4294864fbabdff5a04dcf5b163 already in progress: creating tablet
I20250625 07:52:24.773819 1160 ts_tablet_manager.cc:1918] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.773939 1160 log.cc:1199] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/08bb47266e8b4a0fa5ec613560b8549e
I20250625 07:52:24.774204 1160 ts_tablet_manager.cc:1939] T 08bb47266e8b4a0fa5ec613560b8549e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.774854 1163 tablet_bootstrap.cc:654] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.775380 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 08bb47266e8b4a0fa5ec613560b8549e (table default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]) successfully deleted
I20250625 07:52:24.777784 1705 consensus_queue.cc:1035] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:24.781390 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c61750d1b097487aa364f4b32693709e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.782017 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c61750d1b097487aa364f4b32693709e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.793699 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.793973 1166 tablet_replica.cc:331] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.794150 1166 raft_consensus.cc:2241] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.794292 1166 raft_consensus.cc:2270] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.795168 1166 ts_tablet_manager.cc:1905] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:24.797247 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c1fc4392f58a4d91973506cb0939bb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fc4392f58a4d91973506cb0939bb4c already in progress: creating tablet
W20250625 07:52:24.797528 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 08bb47266e8b4a0fa5ec613560b8549e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08bb47266e8b4a0fa5ec613560b8549e already in progress: creating tablet
I20250625 07:52:24.803059 1163 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.803205 1163 ts_tablet_manager.cc:1397] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.004s
I20250625 07:52:24.803969 1163 raft_consensus.cc:357] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.804147 1163 raft_consensus.cc:383] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.804196 1163 raft_consensus.cc:738] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.804414 1163 consensus_queue.cc:260] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.806771 1166 ts_tablet_manager.cc:1918] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.806932 1166 log.cc:1199] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/21658216da3544a2b8b8f39667782ffb
I20250625 07:52:24.808327 1166 ts_tablet_manager.cc:1939] T 21658216da3544a2b8b8f39667782ffb P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.808920 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a798b2471104d69b814df95edc541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.809123 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c3c180785b64f7d85863c14cfd09ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.810158 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 21658216da3544a2b8b8f39667782ffb (table default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]) successfully deleted
W20250625 07:52:24.810811 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7a798b2471104d69b814df95edc541c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a798b2471104d69b814df95edc541c6 already in progress: creating tablet
W20250625 07:52:24.811453 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5ec29e4294864fbabdff5a04dcf5b163 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ec29e4294864fbabdff5a04dcf5b163 already in progress: creating tablet
I20250625 07:52:24.813582 1160 tablet_replica.cc:331] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.813925 1161 tablet_replica.cc:331] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:24.814177 1161 raft_consensus.cc:2241] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.814499 1160 raft_consensus.cc:2241] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:24.814589 850 consensus_peers.cc:487] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:24.814635 1160 raft_consensus.cc:2270] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.815124 1160 ts_tablet_manager.cc:1905] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.816253 1161 raft_consensus.cc:2270] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.818140 1163 ts_tablet_manager.cc:1428] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250625 07:52:24.818439 1163 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.820067 1163 tablet_bootstrap.cc:654] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.822737 1162 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.822928 1162 ts_tablet_manager.cc:1397] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.082s user 0.026s sys 0.015s
I20250625 07:52:24.824295 1162 raft_consensus.cc:357] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:24.824564 1162 raft_consensus.cc:383] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.824944 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dbc79667956e45a6963ad166c7a8440d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:24.825033 1162 raft_consensus.cc:738] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.825268 1162 consensus_queue.cc:260] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:24.825780 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_891a8459d2ca484ba38e14d7f5768fdb" } new_table_name: "default.Table_d4f6048941564dd399bc7458689264ab" modify_external_catalogs: true
I20250625 07:52:24.827554 1162 ts_tablet_manager.cc:1428] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:24.827816 1162 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:24.828855 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbc79667956e45a6963ad166c7a8440d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.829388 1162 tablet_bootstrap.cc:654] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:24.831053 1687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:52:24.831197 1687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:24.831334 1687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:52:24.831427 1687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:24.833421 1161 ts_tablet_manager.cc:1905] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.837715 1158 tablet_bootstrap.cc:654] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.855787 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6aeac87c40794fa28a9541a027c60bcd" } modify_external_catalogs: true
I20250625 07:52:24.856031 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6aeac87c40794fa28a9541a027c60bcd" } modify_external_catalogs: true
I20250625 07:52:24.857396 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aca9ba5e918840cead995f8ee2abb536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.857971 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aca9ba5e918840cead995f8ee2abb536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.862509 1687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=39255
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:24.862712 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_baabb7e0fd1e496bbe725e5e00574146" } modify_external_catalogs: true
I20250625 07:52:24.862883 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_baabb7e0fd1e496bbe725e5e00574146" } modify_external_catalogs: true
W20250625 07:52:24.863436 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05c040937f7646f5a0fbb98d533250b8 consensus state has no leader
I20250625 07:52:24.863729 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7476f8055aa048e9b8c30b2269f4f2c9" } modify_external_catalogs: true
I20250625 07:52:24.863857 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7476f8055aa048e9b8c30b2269f4f2c9" } modify_external_catalogs: true
I20250625 07:52:24.864531 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8904912af22c440e8d22530ea0c38f62" } new_table_name: "default.Table_4b28648f05f54681a7b82f8429d849ef" modify_external_catalogs: true
I20250625 07:52:24.865183 912 raft_consensus.cc:1273] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250625 07:52:24.865760 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b160757deef94a008c500d961bbb19ab consensus state has no leader
I20250625 07:52:24.865705 1660 consensus_queue.cc:1035] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:24.866034 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_302f6068ad054b38aa3d44e3dd369c8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.866514 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_302f6068ad054b38aa3d44e3dd369c8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.868410 1162 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:24.868409 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 59df0f66116e4dfd9386d107cab0aaf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.868588 1162 ts_tablet_manager.cc:1397] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.003s
I20250625 07:52:24.869593 1162 raft_consensus.cc:357] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.869772 1162 raft_consensus.cc:383] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.869828 1162 raft_consensus.cc:738] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.870127 1162 consensus_queue.cc:260] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:24.872072 713 consensus_peers.cc:487] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:24.874498 1162 ts_tablet_manager.cc:1428] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250625 07:52:24.876847 1660 raft_consensus.cc:491] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.877015 1163 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:24.876971 1660 raft_consensus.cc:513] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.877130 1163 ts_tablet_manager.cc:1397] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.059s user 0.008s sys 0.008s
I20250625 07:52:24.877331 1161 ts_tablet_manager.cc:1918] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.877458 1161 log.cc:1199] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4c3c180785b64f7d85863c14cfd09ccc
I20250625 07:52:24.877557 1660 leader_election.cc:290] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.877884 1161 ts_tablet_manager.cc:1939] T 4c3c180785b64f7d85863c14cfd09ccc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:24.878816 713 leader_election.cc:343] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:24.879002 1705 raft_consensus.cc:491] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.879125 1705 raft_consensus.cc:513] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.879676 1705 leader_election.cc:290] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.880189 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1422987deca41a79f8db666d6961643" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:24.880724 851 leader_election.cc:343] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.881045 1162 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:24.883102 1163 raft_consensus.cc:357] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.883644 1163 raft_consensus.cc:383] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:24.883888 1163 raft_consensus.cc:738] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:24.884313 1163 consensus_queue.cc:260] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.886054 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d1770f6d5d1d4899a66ec23ab4eadc35
I20250625 07:52:24.886186 1162 tablet_bootstrap.cc:654] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:24.886755 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.886956 1166 tablet_replica.cc:331] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.887115 1166 raft_consensus.cc:2241] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:24.887459 1166 raft_consensus.cc:2270] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.887993 1166 ts_tablet_manager.cc:1905] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.888293 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.889367 1163 ts_tablet_manager.cc:1428] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:24.889796 1163 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:24.890197 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet a630f8804bbf41de86b08cde419643ff (table default.Table_929703cfa29e467c9090ff84a8b2cec1 [id=cd9d83bd28e5422188207277162018a1]). Expected version 2 got 0
I20250625 07:52:24.890455 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c8c015c202544914b74f1e245b496f82 (table default.Table_2b519d271b96497dafe0b274a8755066 [id=4a285d0804ea403bb60aa49bfde4bd9a]). Expected version 1 got 0
I20250625 07:52:24.892292 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
I20250625 07:52:24.894410 1166 ts_tablet_manager.cc:1918] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:24.894526 1166 log.cc:1199] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/d1770f6d5d1d4899a66ec23ab4eadc35
I20250625 07:52:24.894868 1166 ts_tablet_manager.cc:1939] T d1770f6d5d1d4899a66ec23ab4eadc35 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.896845 747 tablet_service.cc:1515] Processing DeleteTablet for tablet ba24bbd0c4694930978a7750f07601d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.897053 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.897472 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.898018 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]) successfully deleted
I20250625 07:52:24.898514 1166 tablet_replica.cc:331] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.878090 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10c7f4b4187c428bbec136c85219da4d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.898809 1166 raft_consensus.cc:2241] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.885682 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1422987deca41a79f8db666d6961643" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.899061 1166 raft_consensus.cc:2270] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:24.899137 713 leader_election.cc:343] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.899269 713 leader_election.cc:304] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:24.899494 1543 raft_consensus.cc:2747] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.900225 1166 ts_tablet_manager.cc:1905] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.901743 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 4c3c180785b64f7d85863c14cfd09ccc (table default.Table_53870f55fcc5414ab6e8ccb92db24344 [id=0a52421cfae54a7981ba5bf06b8efead]) successfully deleted
I20250625 07:52:24.888636 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet d1770f6d5d1d4899a66ec23ab4eadc35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.904340 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
I20250625 07:52:24.904707 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a [id=b0f32fb69f33447db7620299affa88c8]). Expected version 3 got 0
I20250625 07:52:24.905848 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]). Expected version 4 got 0
I20250625 07:52:24.907269 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.907518 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.909300 755 tablet_service.cc:1468] Processing CreateTablet for tablet a8a668fb466840e0a41749a4db9a1554 (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.909742 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a668fb466840e0a41749a4db9a1554. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.909971 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a8a668fb466840e0a41749a4db9a1554 (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.911011 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a668fb466840e0a41749a4db9a1554. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.915779 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 487705c4b9ae4ab097771a69f03f05dc (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.916222 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487705c4b9ae4ab097771a69f03f05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.919901 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef2eb85358b414c9755f81f97beaddb (DEFAULT_TABLE table=default.Table_b2cb8c1f912f43bfad73814acbecbc34 [id=52d1cb1ab4ab4cd4b08dccb0ec5c134c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.920282 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef2eb85358b414c9755f81f97beaddb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.921290 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96c58bffafa146ffacba18cf615c74b3 consensus state has no leader
I20250625 07:52:24.925005 747 tablet_service.cc:1468] Processing CreateTablet for tablet 231ced6803944e24829caf7853da017d (DEFAULT_TABLE table=default.Table_ac08bcc7329545ba8ba93e26d1c810f2 [id=a01e96ae60c9403ba16dc7c8313d7e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.925387 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231ced6803944e24829caf7853da017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.928329 755 tablet_service.cc:1468] Processing CreateTablet for tablet 487705c4b9ae4ab097771a69f03f05dc (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.928674 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487705c4b9ae4ab097771a69f03f05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.929001 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:24.929101 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 231ced6803944e24829caf7853da017d (DEFAULT_TABLE table=default.Table_ac08bcc7329545ba8ba93e26d1c810f2 [id=a01e96ae60c9403ba16dc7c8313d7e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.929509 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231ced6803944e24829caf7853da017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.930733 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e1eb60c1e8c146a596391600eaffc9d7" } new_table_name: "default.Table_824016809113454d9d932a22a8d60512" modify_external_catalogs: true
I20250625 07:52:24.930860 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.932252 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.933171 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 (DEFAULT_TABLE table=default.Table_0c7a05e580684fb2811d89a355b0a26e [id=983d2b24dc2f407c94dda3d07cfd075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.933524 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.934182 744 tablet_service.cc:1468] Processing CreateTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 (DEFAULT_TABLE table=default.Table_0c7a05e580684fb2811d89a355b0a26e [id=983d2b24dc2f407c94dda3d07cfd075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.934522 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.937418 742 tablet_service.cc:1468] Processing CreateTablet for tablet 926879d7917644aabd289597e47eca55 (DEFAULT_TABLE table=default.Table_d85c53889892469d956fb4d28e74e2b0 [id=4867f27aca81474c9221767f29592358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.937785 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926879d7917644aabd289597e47eca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.938484 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 26db0effa17b4cacb1450a01f9631ddf
I20250625 07:52:24.939060 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 26db0effa17b4cacb1450a01f9631ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.940009 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 26db0effa17b4cacb1450a01f9631ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.940766 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 926879d7917644aabd289597e47eca55 (DEFAULT_TABLE table=default.Table_d85c53889892469d956fb4d28e74e2b0 [id=4867f27aca81474c9221767f29592358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.941201 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:24.941431 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926879d7917644aabd289597e47eca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.941814 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:24.943850 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:24.944849 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f001237d203b42898ff32eff812b5557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.945429 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f001237d203b42898ff32eff812b5557 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.948122 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 (DEFAULT_TABLE table=default.Table_1b252524928e4ed3a800c849b1ffd8e0 [id=441b3ce49b5e44b98319fd850fcaa6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.948282 1166 ts_tablet_manager.cc:1918] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.948401 1166 log.cc:1199] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:24.948608 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168f1a13533045d89ba1f07a72d4ff63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.948734 1166 ts_tablet_manager.cc:1939] T ba24bbd0c4694930978a7750f07601d9 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:24.949596 1166 tablet_replica.cc:331] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:24.949774 1166 raft_consensus.cc:2241] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.950129 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet ba24bbd0c4694930978a7750f07601d9 (table default.Table_ce3de32cf03042d6990e81e54f461669 [id=30de507c3fb344ecb071e9def7dcf119]) successfully deleted
I20250625 07:52:24.893354 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:24.905161 850 leader_election.cc:343] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.950482 850 leader_election.cc:304] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:24.950814 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 45652.8us
I20250625 07:52:24.950875 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef2eb85358b414c9755f81f97beaddb (DEFAULT_TABLE table=default.Table_b2cb8c1f912f43bfad73814acbecbc34 [id=52d1cb1ab4ab4cd4b08dccb0ec5c134c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.907085 1705 raft_consensus.cc:491] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:24.951932 1729 raft_consensus.cc:2747] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:24.952107 1705 raft_consensus.cc:513] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:24.952960 1705 leader_election.cc:290] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:24.953249 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.953763 891 tablet_service.cc:1468] Processing CreateTablet for tablet a8a668fb466840e0a41749a4db9a1554 (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.953847 1166 raft_consensus.cc:2270] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.954191 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a668fb466840e0a41749a4db9a1554. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.954836 1166 ts_tablet_manager.cc:1905] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.913735 1160 ts_tablet_manager.cc:1918] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.955942 1160 log.cc:1199] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:24.956290 1160 ts_tablet_manager.cc:1939] T ba24bbd0c4694930978a7750f07601d9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.957024 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef2eb85358b414c9755f81f97beaddb (DEFAULT_TABLE table=default.Table_b2cb8c1f912f43bfad73814acbecbc34 [id=52d1cb1ab4ab4cd4b08dccb0ec5c134c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.957384 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef2eb85358b414c9755f81f97beaddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.958209 1160 tablet_replica.cc:331] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:24.958489 1160 raft_consensus.cc:2241] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.958657 1160 raft_consensus.cc:2270] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.959408 1160 ts_tablet_manager.cc:1905] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.959651 1161 tablet_replica.cc:331] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:24.959599 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98c9cffe906a4a99bedb93374dab75e4" } modify_external_catalogs: true
I20250625 07:52:24.959810 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_98c9cffe906a4a99bedb93374dab75e4" } modify_external_catalogs: true
I20250625 07:52:24.960036 1161 raft_consensus.cc:2241] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:24.960268 1161 raft_consensus.cc:2270] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:24.961169 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.945839 1761 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:24.961795 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8c015c202544914b74f1e245b496f82" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:24.962873 895 tablet_service.cc:1468] Processing CreateTablet for tablet 487705c4b9ae4ab097771a69f03f05dc (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.963264 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487705c4b9ae4ab097771a69f03f05dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.963294 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 96c58bffafa146ffacba18cf615c74b3 (DEFAULT_TABLE table=default.Table_4b28648f05f54681a7b82f8429d849ef [id=3090fa0ae0cd40b2ac4cd67991b7b08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.963708 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c58bffafa146ffacba18cf615c74b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:24.965900 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a396bde4b5964697ad0090eab777ba0e consensus state has no leader
I20250625 07:52:24.966869 883 tablet_service.cc:1468] Processing CreateTablet for tablet 231ced6803944e24829caf7853da017d (DEFAULT_TABLE table=default.Table_ac08bcc7329545ba8ba93e26d1c810f2 [id=a01e96ae60c9403ba16dc7c8313d7e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.967262 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231ced6803944e24829caf7853da017d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.969105 1161 ts_tablet_manager.cc:1905] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:24.970103 1160 ts_tablet_manager.cc:1918] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:24.970180 851 leader_election.cc:343] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:24.970263 1160 log.cc:1199] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b1422987deca41a79f8db666d6961643
I20250625 07:52:24.970577 1160 ts_tablet_manager.cc:1939] T b1422987deca41a79f8db666d6961643 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:24.977042 893 tablet_service.cc:1468] Processing CreateTablet for tablet 926879d7917644aabd289597e47eca55 (DEFAULT_TABLE table=default.Table_d85c53889892469d956fb4d28e74e2b0 [id=4867f27aca81474c9221767f29592358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.981081 886 tablet_service.cc:1468] Processing CreateTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 (DEFAULT_TABLE table=default.Table_0c7a05e580684fb2811d89a355b0a26e [id=983d2b24dc2f407c94dda3d07cfd075b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.981606 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926879d7917644aabd289597e47eca55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.981731 755 tablet_service.cc:1468] Processing CreateTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 (DEFAULT_TABLE table=default.Table_1b252524928e4ed3a800c849b1ffd8e0 [id=441b3ce49b5e44b98319fd850fcaa6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.982161 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168f1a13533045d89ba1f07a72d4ff63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.982633 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.981621 742 tablet_service.cc:1468] Processing CreateTablet for tablet 96c58bffafa146ffacba18cf615c74b3 (DEFAULT_TABLE table=default.Table_4b28648f05f54681a7b82f8429d849ef [id=3090fa0ae0cd40b2ac4cd67991b7b08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.982824 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 26db0effa17b4cacb1450a01f9631ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:24.984913 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8c015c202544914b74f1e245b496f82" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:24.985287 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c58bffafa146ffacba18cf615c74b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.985852 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet ba24bbd0c4694930978a7750f07601d9 (table default.Table_ce3de32cf03042d6990e81e54f461669 [id=30de507c3fb344ecb071e9def7dcf119]) successfully deleted
W20250625 07:52:24.986131 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c1fc4392f58a4d91973506cb0939bb4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1fc4392f58a4d91973506cb0939bb4c already in progress: creating tablet
I20250625 07:52:24.986624 884 tablet_service.cc:1468] Processing CreateTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 (DEFAULT_TABLE table=default.Table_1b252524928e4ed3a800c849b1ffd8e0 [id=441b3ce49b5e44b98319fd850fcaa6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.988212 892 tablet_service.cc:1468] Processing CreateTablet for tablet 96c58bffafa146ffacba18cf615c74b3 (DEFAULT_TABLE table=default.Table_4b28648f05f54681a7b82f8429d849ef [id=3090fa0ae0cd40b2ac4cd67991b7b08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:24.993569 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b47b10ad384d42129c9bf8891ecd6673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.994186 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b47b10ad384d42129c9bf8891ecd6673 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.994944 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96c58bffafa146ffacba18cf615c74b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:24.995213 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5b6911bd0ce04836b8e4215c59c5847e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:24.995738 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b6911bd0ce04836b8e4215c59c5847e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:24.997308 1166 ts_tablet_manager.cc:1918] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:24.997460 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8f7240577a874a59b3268ff7f0c1beec
I20250625 07:52:24.999001 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a [id=b0f32fb69f33447db7620299affa88c8]). Expected version 3 got 0
I20250625 07:52:24.999312 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f7240577a874a59b3268ff7f0c1beec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.000382 1166 log.cc:1199] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7f196473315441c6bf8b5173ed46f833
I20250625 07:52:25.000794 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 168f1a13533045d89ba1f07a72d4ff63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.001134 1166 ts_tablet_manager.cc:1939] T 7f196473315441c6bf8b5173ed46f833 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.003031 1166 tablet_replica.cc:331] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.003043 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7f196473315441c6bf8b5173ed46f833 (table default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]) successfully deleted
I20250625 07:52:25.003217 1166 raft_consensus.cc:2241] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.003371 1166 raft_consensus.cc:2270] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.003671 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f7240577a874a59b3268ff7f0c1beec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.003934 1158 tablet_bootstrap.cc:492] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:25.004140 1158 ts_tablet_manager.cc:1397] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.350s user 0.003s sys 0.007s
I20250625 07:52:25.005043 1166 ts_tablet_manager.cc:1905] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.006449 1158 raft_consensus.cc:357] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.006738 1158 raft_consensus.cc:383] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.006822 1158 raft_consensus.cc:738] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.007098 1158 consensus_queue.cc:260] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.007673 1158 ts_tablet_manager.cc:1428] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:25.012269 1158 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:25.015691 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.015785 1158 tablet_bootstrap.cc:654] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.021826 1160 tablet_replica.cc:331] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.022094 1160 raft_consensus.cc:2241] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:25.022254 850 leader_election.cc:343] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.022701 1160 raft_consensus.cc:2270] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.023269 1160 ts_tablet_manager.cc:1905] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.026907 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0452e73865964f709310fd22ac39a5f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.033270 1166 ts_tablet_manager.cc:1918] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.033761 1166 log.cc:1199] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a7cb2bcdd07c42e680eb4f155212423f
I20250625 07:52:25.034277 1166 ts_tablet_manager.cc:1939] T a7cb2bcdd07c42e680eb4f155212423f P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.035533 1166 tablet_replica.cc:331] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.036679 1166 raft_consensus.cc:2241] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.037175 1166 raft_consensus.cc:2270] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.038280 1166 ts_tablet_manager.cc:1905] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.041556 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f3fa68b5cb55490083020c843fe1108a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.041900 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a7cb2bcdd07c42e680eb4f155212423f (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
W20250625 07:52:25.042256 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3fa68b5cb55490083020c843fe1108a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.044153 1760 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:25.046985 1764 tablet.cc:1722] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:25.050496 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cc08960a4b664c009b763304b167cf6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.051090 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc08960a4b664c009b763304b167cf6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.052561 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f666c9ec62da45a5ba3f010f96476b30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.053118 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f666c9ec62da45a5ba3f010f96476b30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.056706 850 leader_election.cc:304] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.069162 1161 ts_tablet_manager.cc:1918] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.069326 1161 log.cc:1199] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/59df0f66116e4dfd9386d107cab0aaf6
I20250625 07:52:25.071883 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f9279117557549b3826d8056f0d22581" } new_table_name: "default.Table_5090762d485f41399e8ac22e3317f483" modify_external_catalogs: true
I20250625 07:52:25.071696 1161 ts_tablet_manager.cc:1939] T 59df0f66116e4dfd9386d107cab0aaf6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:25.073447 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d29d1b5f858f41d590392cfd6ec5c0d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.074280 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 59df0f66116e4dfd9386d107cab0aaf6 (table default.Table_6939d0472b4b4e99b00dcaafdf3ba7a1 [id=4793737aa6374faabd04844941df9f1a]) successfully deleted
I20250625 07:52:25.076268 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d841dee5b83d4242b8644949082fa7a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.076965 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f666c9ec62da45a5ba3f010f96476b30" } new_table_name: "default.Table_e753ff3879ec40f58838c14425717b90" modify_external_catalogs: true
I20250625 07:52:25.077013 1161 tablet_replica.cc:331] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:25.078218 1687 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:25.078773 1705 raft_consensus.cc:2747] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.079726 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cdfa29007d814e609da9a03c52f558f2" } new_table_name: "default.Table_8f7140aa42f54e6b80e1e6ce1202eb2c" modify_external_catalogs: true
I20250625 07:52:25.082422 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:25.085376 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d841dee5b83d4242b8644949082fa7a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:25.086773 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d29d1b5f858f41d590392cfd6ec5c0d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.088308 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5" } modify_external_catalogs: true
I20250625 07:52:25.088945 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5aa6c605c6864a9b88a9993a6ce0a1e5" } modify_external_catalogs: true
I20250625 07:52:25.090077 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e13b99b01ed747aeaf4c7005e326366a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.090545 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f54fa729d3754d7eb7e0528a733d40b6 has no consensus state
W20250625 07:52:25.090617 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e13b99b01ed747aeaf4c7005e326366a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:25.091784 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9eaab8f86bdc45cab8645cd5a6e3489c consensus state has no leader
W20250625 07:52:25.092840 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20d6334831b942399273d8d1314d7908 consensus state has no leader
I20250625 07:52:25.093870 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a [id=b0f32fb69f33447db7620299affa88c8] alter complete (version 3)
I20250625 07:52:25.096558 1158 tablet_bootstrap.cc:492] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:25.101289 1651 raft_consensus.cc:491] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.101454 1651 raft_consensus.cc:513] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.102157 1651 leader_election.cc:290] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.102881 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "366d3e6fcc9d49be8fbd2cd070632861" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.103487 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 78847.4us
I20250625 07:52:25.105091 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b1422987deca41a79f8db666d6961643 (table default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]) successfully deleted
I20250625 07:52:25.105082 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "366d3e6fcc9d49be8fbd2cd070632861" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.105376 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7a798b2471104d69b814df95edc541c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a798b2471104d69b814df95edc541c6 already in progress: creating tablet
I20250625 07:52:25.106868 1161 raft_consensus.cc:2241] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.107173 1161 raft_consensus.cc:2270] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.109489 1761 tablet.cc:1722] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:25.096788 1158 ts_tablet_manager.cc:1397] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.000s
I20250625 07:52:25.111462 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 3ab38e0cdcd14f209e6877f2e4a46e98 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:25.112121 1162 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:25.112099 1158 raft_consensus.cc:357] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.112339 1162 ts_tablet_manager.cc:1397] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.231s user 0.023s sys 0.007s
I20250625 07:52:25.112399 1651 raft_consensus.cc:491] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.112805 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f7240577a874a59b3268ff7f0c1beec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:25.112963 851 leader_election.cc:343] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:25.113134 850 leader_election.cc:343] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.113318 1162 raft_consensus.cc:357] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.113659 1162 raft_consensus.cc:383] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.113618 850 leader_election.cc:304] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.113883 1705 raft_consensus.cc:2747] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.113777 1162 raft_consensus.cc:738] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.114326 1162 consensus_queue.cc:260] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.117089 1651 raft_consensus.cc:513] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.116231 1158 raft_consensus.cc:383] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.117583 1158 raft_consensus.cc:738] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.118585 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2] alter complete (version 4)
I20250625 07:52:25.117780 1158 consensus_queue.cc:260] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.119416 1162 ts_tablet_manager.cc:1428] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 07:52:25.117192 1161 ts_tablet_manager.cc:1905] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.123353 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c2ec45c7324942d39b6da745f3ca5e1f" } new_table_name: "default.Table_a2fb991094f545cfb3a7f7aefb03d8a5" modify_external_catalogs: true
I20250625 07:52:25.124807 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.127183 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b620bb670cd94dce999cbe1dabc6125a (DEFAULT_TABLE table=default.Table_aca9ba5e918840cead995f8ee2abb536 [id=b5682b29a7264b6293677aaa101fb900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.127548 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b620bb670cd94dce999cbe1dabc6125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.128091 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6a6fab215b63465a9e48738d3f27651e
I20250625 07:52:25.129392 1651 leader_election.cc:290] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.129973 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a6fab215b63465a9e48738d3f27651e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.129976 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_62b2ccb944794418a0e39d5874f5ef7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.130331 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a630f8804bbf41de86b08cde419643ff" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.130466 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62b2ccb944794418a0e39d5874f5ef7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.131436 1162 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:25.132503 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b43beebf15d44d2a37382c52d1c35fb" } modify_external_catalogs: true
I20250625 07:52:25.132862 1163 tablet_bootstrap.cc:654] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.133064 1162 tablet_bootstrap.cc:654] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.133178 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a630f8804bbf41de86b08cde419643ff" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:25.133239 1160 ts_tablet_manager.cc:1918] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:25.132747 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b43beebf15d44d2a37382c52d1c35fb" } modify_external_catalogs: true
I20250625 07:52:25.135399 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a6fab215b63465a9e48738d3f27651e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:25.135571 850 leader_election.cc:343] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.135993 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a6fab215b63465a9e48738d3f27651e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:25.136224 851 leader_election.cc:343] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.137692 1166 ts_tablet_manager.cc:1918] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.137830 1166 log.cc:1199] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:25.138167 1166 ts_tablet_manager.cc:1939] T 10c7f4b4187c428bbec136c85219da4d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.139110 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 10c7f4b4187c428bbec136c85219da4d (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
I20250625 07:52:25.139442 1166 tablet_replica.cc:331] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.139544 1160 log.cc:1199] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d1770f6d5d1d4899a66ec23ab4eadc35
I20250625 07:52:25.140053 1166 raft_consensus.cc:2241] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.140233 1166 raft_consensus.cc:2270] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.140379 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32" } modify_external_catalogs: true
I20250625 07:52:25.140573 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f2546b9a23ca4ad1aa2c45bf380a7a32" } modify_external_catalogs: true
I20250625 07:52:25.140947 1166 ts_tablet_manager.cc:1905] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.143780 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_16fdbb4827074529aaa4f9a805a01468" } modify_external_catalogs: true
I20250625 07:52:25.143981 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_16fdbb4827074529aaa4f9a805a01468" } modify_external_catalogs: true
I20250625 07:52:25.144840 1651 raft_consensus.cc:491] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.142326 851 leader_election.cc:304] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.145341 1160 ts_tablet_manager.cc:1939] T d1770f6d5d1d4899a66ec23ab4eadc35 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.146327 1160 tablet_replica.cc:331] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.146653 1705 raft_consensus.cc:2747] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.146799 1651 raft_consensus.cc:513] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.147169 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]) successfully deleted
I20250625 07:52:25.147749 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0bbd21d4aa41b2b92473bffe7dc05a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:25.148322 1651 leader_election.cc:290] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:25.148514 851 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.148677 1160 raft_consensus.cc:2241] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.148912 1160 raft_consensus.cc:2270] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.149183 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0bbd21d4aa41b2b92473bffe7dc05a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.149751 850 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.149873 850 leader_election.cc:304] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.150099 1651 raft_consensus.cc:2747] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:25.152436 583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477): Couldn't send request to peer 00a3faeffaa4422ea73b98ee74768a4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:25.152774 1160 ts_tablet_manager.cc:1905] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.154899 1166 ts_tablet_manager.cc:1918] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.157336 1166 log.cc:1199] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/802c22a6346442c88abbeb4b9ef20c57
I20250625 07:52:25.158100 1166 ts_tablet_manager.cc:1939] T 802c22a6346442c88abbeb4b9ef20c57 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.159343 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 802c22a6346442c88abbeb4b9ef20c57 (table default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]) successfully deleted
W20250625 07:52:25.160511 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 896419bd407f43e1a0a361330ff9821b consensus state has no leader
W20250625 07:52:25.161464 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0657163b9a0d4f7f99f57f7fd1f32462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0657163b9a0d4f7f99f57f7fd1f32462 already in progress: creating tablet
I20250625 07:52:25.161577 1158 ts_tablet_manager.cc:1428] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20250625 07:52:25.161918 1158 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:25.165217 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 26db0effa17b4cacb1450a01f9631ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26db0effa17b4cacb1450a01f9631ddf already in progress: creating tablet
W20250625 07:52:25.166554 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 70094e6c7e644f62bf5ef04dcfb4c380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70094e6c7e644f62bf5ef04dcfb4c380 already in progress: creating tablet
I20250625 07:52:25.167675 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2fb37c46511c4963a2b384669ec6331c" } new_table_name: "default.Table_965ace13b8ae452ab1e388fe37111c9a" modify_external_catalogs: true
I20250625 07:52:25.168068 886 tablet_service.cc:1468] Processing CreateTablet for tablet b620bb670cd94dce999cbe1dabc6125a (DEFAULT_TABLE table=default.Table_aca9ba5e918840cead995f8ee2abb536 [id=b5682b29a7264b6293677aaa101fb900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.168495 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b620bb670cd94dce999cbe1dabc6125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.168848 755 tablet_service.cc:1468] Processing CreateTablet for tablet b620bb670cd94dce999cbe1dabc6125a (DEFAULT_TABLE table=default.Table_aca9ba5e918840cead995f8ee2abb536 [id=b5682b29a7264b6293677aaa101fb900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.169214 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b620bb670cd94dce999cbe1dabc6125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.171504 1687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 07:52:25.175438 1158 tablet_bootstrap.cc:654] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:25.176549 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8f7240577a874a59b3268ff7f0c1beec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f7240577a874a59b3268ff7f0c1beec already in progress: creating tablet
I20250625 07:52:25.177268 1166 tablet_replica.cc:331] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.177678 1166 raft_consensus.cc:2241] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.178115 1166 raft_consensus.cc:2270] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.178876 1160 ts_tablet_manager.cc:1918] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.179006 1160 log.cc:1199] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3bed3f04c60249aaa389c26788974f58
I20250625 07:52:25.179342 1160 ts_tablet_manager.cc:1939] T 3bed3f04c60249aaa389c26788974f58 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.179819 1166 ts_tablet_manager.cc:1905] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.180547 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3bed3f04c60249aaa389c26788974f58 (table default.Table_a6db9072062d4e5895af0868d99ac31b [id=8e22a511cbc44f8996d35e514dfda631]) successfully deleted
I20250625 07:52:25.181587 1163 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.181720 1163 ts_tablet_manager.cc:1397] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.292s user 0.008s sys 0.008s
I20250625 07:52:25.182368 1163 raft_consensus.cc:357] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.182544 1163 raft_consensus.cc:383] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.182595 1163 raft_consensus.cc:738] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.182811 1163 consensus_queue.cc:260] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.183517 1163 ts_tablet_manager.cc:1428] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:25.183775 1163 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.187492 1160 tablet_replica.cc:331] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.187686 1160 raft_consensus.cc:2241] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.187876 1160 raft_consensus.cc:2270] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.189132 1160 ts_tablet_manager.cc:1905] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.190851 755 tablet_service.cc:1468] Processing CreateTablet for tablet b927ec564efa4ff080d22a673abb9b57 (DEFAULT_TABLE table=default.Table_c61750d1b097487aa364f4b32693709e [id=73b04ab6d8d64bde8af734e2d18dd9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.191238 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927ec564efa4ff080d22a673abb9b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.191496 895 tablet_service.cc:1468] Processing CreateTablet for tablet b927ec564efa4ff080d22a673abb9b57 (DEFAULT_TABLE table=default.Table_c61750d1b097487aa364f4b32693709e [id=73b04ab6d8d64bde8af734e2d18dd9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.191828 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927ec564efa4ff080d22a673abb9b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.192987 742 tablet_service.cc:1468] Processing CreateTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 (DEFAULT_TABLE table=default.Table_2e1e160bec8647f2b41862f84d3e4817 [id=7f48c4b1013f4831a97566298978262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.193311 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8968da295c4b7fb6d83b84935d27d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.194805 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe (DEFAULT_TABLE table=default.Table_dc8c4809369446dc9c12a878a16e6c48 [id=946044b25bd24ac6b79d561f9ed848d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.195123 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.197217 1163 tablet_bootstrap.cc:654] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.197746 743 tablet_service.cc:1468] Processing CreateTablet for tablet b6661b294489423ab06fa7c6988b75f7 (DEFAULT_TABLE table=default.Table_a4a61d6fd2514d88989f611b35c12baa [id=f07bd577df8545a7a31a2300e2662d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.198227 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b294489423ab06fa7c6988b75f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.198912 884 tablet_service.cc:1468] Processing CreateTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 (DEFAULT_TABLE table=default.Table_2e1e160bec8647f2b41862f84d3e4817 [id=7f48c4b1013f4831a97566298978262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.199244 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8968da295c4b7fb6d83b84935d27d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.200955 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe (DEFAULT_TABLE table=default.Table_dc8c4809369446dc9c12a878a16e6c48 [id=946044b25bd24ac6b79d561f9ed848d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.201339 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.205050 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca2f298aefc42e890d0ae48f26ba1ce (DEFAULT_TABLE table=default.Table_7a254ead6d2e4a3e9ede91b319970b00 [id=2514253c25be4a27b4ff989ed1d62c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.205387 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca2f298aefc42e890d0ae48f26ba1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.207326 1166 ts_tablet_manager.cc:1918] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.207441 1166 log.cc:1199] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/31942f0998bb4118a91f476325851d5c
I20250625 07:52:25.207736 1166 ts_tablet_manager.cc:1939] T 31942f0998bb4118a91f476325851d5c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.208734 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 31942f0998bb4118a91f476325851d5c (table default.Table_a28f5f4927f34e03b2ea4784b65cfe43 [id=7686e223b6fb47e79ecb2257550a64b5]) successfully deleted
I20250625 07:52:25.209749 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 5cbace0f99a44800bab1de5e38eb5112 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:25.210266 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:25.210263 747 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca2f298aefc42e890d0ae48f26ba1ce (DEFAULT_TABLE table=default.Table_7a254ead6d2e4a3e9ede91b319970b00 [id=2514253c25be4a27b4ff989ed1d62c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.210636 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca2f298aefc42e890d0ae48f26ba1ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:25.210721 593 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 89c703dc3249408faeb522cef0248111 consensus state has no leader
I20250625 07:52:25.211052 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6e438577c44d4924bbfad6fa2cc3beaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.211560 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e438577c44d4924bbfad6fa2cc3beaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.212636 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aca9ba5e918840cead995f8ee2abb536" } modify_external_catalogs: true
I20250625 07:52:25.212854 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aca9ba5e918840cead995f8ee2abb536" } modify_external_catalogs: true
W20250625 07:52:25.216161 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6a6fab215b63465a9e48738d3f27651e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a6fab215b63465a9e48738d3f27651e already in progress: creating tablet
I20250625 07:52:25.218722 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fd9677bf99f041c2ade65c94f0170609
I20250625 07:52:25.218730 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0a0fe49e81af4d8eb81271cf7b8057d1 (DEFAULT_TABLE table=default.Table_0d7d47bc82394837aff28ef6e5d70c8f [id=1fc1c8718d70481c9e5575db1298f4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.218968 891 tablet_service.cc:1468] Processing CreateTablet for tablet 0a0fe49e81af4d8eb81271cf7b8057d1 (DEFAULT_TABLE table=default.Table_0d7d47bc82394837aff28ef6e5d70c8f [id=1fc1c8718d70481c9e5575db1298f4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.219092 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0fe49e81af4d8eb81271cf7b8057d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.219321 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0fe49e81af4d8eb81271cf7b8057d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.219959 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4f3b6dc279984d7fba83a6308fbc0fba
I20250625 07:52:25.220628 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0e4519b1fbf34ca8b33655f4d402aafc
I20250625 07:52:25.220847 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.221230 883 tablet_service.cc:1515] Processing DeleteTablet for tablet fd9677bf99f041c2ade65c94f0170609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.221644 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.222424 743 tablet_service.cc:1515] Processing DeleteTablet for tablet fd9677bf99f041c2ade65c94f0170609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.222537 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.222783 750 tablet_service.cc:1468] Processing CreateTablet for tablet 799be9c1ae93465d9f70fc960d50bec7 (DEFAULT_TABLE table=default.Table_06029eba0bd241169e60166d8be6654b [id=622c472d18584bdd98fa4dd8d35ff41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.222858 883 tablet_service.cc:1468] Processing CreateTablet for tablet 799be9c1ae93465d9f70fc960d50bec7 (DEFAULT_TABLE table=default.Table_06029eba0bd241169e60166d8be6654b [id=622c472d18584bdd98fa4dd8d35ff41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.223117 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799be9c1ae93465d9f70fc960d50bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.223130 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799be9c1ae93465d9f70fc960d50bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.224179 743 tablet_service.cc:1468] Processing CreateTablet for tablet 745cd19f3b964febbba91f9bf95755c8 (DEFAULT_TABLE table=default.Table_c2765164e15a4bc8adeae4612f437ce2 [id=0cd6f94f154e4eb3a0855b42717e72b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.224524 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745cd19f3b964febbba91f9bf95755c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.228924 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.229693 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d45fec10075f4eb898fd3cb99709c2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.230285 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45fec10075f4eb898fd3cb99709c2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.231675 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d294d1a1c5f749e9ae13e0bb5e4f404e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.232156 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d294d1a1c5f749e9ae13e0bb5e4f404e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:25.233421 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fd9677bf99f041c2ade65c94f0170609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd9677bf99f041c2ade65c94f0170609 already in progress: creating tablet
I20250625 07:52:25.233765 1166 tablet_replica.cc:331] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.233976 1166 raft_consensus.cc:2241] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:25.234314 1166 raft_consensus.cc:2270] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.237600 1163 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.237736 1163 ts_tablet_manager.cc:1397] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20250625 07:52:25.193812 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe (DEFAULT_TABLE table=default.Table_dc8c4809369446dc9c12a878a16e6c48 [id=946044b25bd24ac6b79d561f9ed848d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.188674 1161 ts_tablet_manager.cc:1918] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:25.238662 1161 log.cc:1199] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/d1770f6d5d1d4899a66ec23ab4eadc35
I20250625 07:52:25.195277 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca2f298aefc42e890d0ae48f26ba1ce (DEFAULT_TABLE table=default.Table_7a254ead6d2e4a3e9ede91b319970b00 [id=2514253c25be4a27b4ff989ed1d62c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.239032 1161 ts_tablet_manager.cc:1939] T d1770f6d5d1d4899a66ec23ab4eadc35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:25.196971 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b6661b294489423ab06fa7c6988b75f7 (DEFAULT_TABLE table=default.Table_a4a61d6fd2514d88989f611b35c12baa [id=f07bd577df8545a7a31a2300e2662d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.217929 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0a0fe49e81af4d8eb81271cf7b8057d1 (DEFAULT_TABLE table=default.Table_0d7d47bc82394837aff28ef6e5d70c8f [id=1fc1c8718d70481c9e5575db1298f4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.223711 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 745cd19f3b964febbba91f9bf95755c8 (DEFAULT_TABLE table=default.Table_c2765164e15a4bc8adeae4612f437ce2 [id=0cd6f94f154e4eb3a0855b42717e72b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.223846 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet fd9677bf99f041c2ade65c94f0170609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.224069 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 799be9c1ae93465d9f70fc960d50bec7 (DEFAULT_TABLE table=default.Table_06029eba0bd241169e60166d8be6654b [id=622c472d18584bdd98fa4dd8d35ff41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.240108 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet d1770f6d5d1d4899a66ec23ab4eadc35 (table default.Table_6aeac87c40794fa28a9541a027c60bcd [id=58c1b09d38bc456280bf241369bef00a]) successfully deleted
I20250625 07:52:25.224152 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f3b6dc279984d7fba83a6308fbc0fba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.224215 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.189921 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b927ec564efa4ff080d22a673abb9b57 (DEFAULT_TABLE table=default.Table_c61750d1b097487aa364f4b32693709e [id=73b04ab6d8d64bde8af734e2d18dd9ce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:25.240775 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0657163b9a0d4f7f99f57f7fd1f32462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0657163b9a0d4f7f99f57f7fd1f32462 already in progress: creating tablet
W20250625 07:52:25.241468 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 26db0effa17b4cacb1450a01f9631ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26db0effa17b4cacb1450a01f9631ddf already in progress: creating tablet
I20250625 07:52:25.238426 1163 raft_consensus.cc:357] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:25.242064 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 70094e6c7e644f62bf5ef04dcfb4c380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70094e6c7e644f62bf5ef04dcfb4c380 already in progress: creating tablet
I20250625 07:52:25.242110 1163 raft_consensus.cc:383] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.242522 1163 raft_consensus.cc:738] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.243738 1163 consensus_queue.cc:260] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.247321 1162 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:25.247509 1162 ts_tablet_manager.cc:1397] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.116s user 0.016s sys 0.018s
I20250625 07:52:25.248291 1162 raft_consensus.cc:357] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.248525 1162 raft_consensus.cc:383] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.248631 1162 raft_consensus.cc:738] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.249198 891 tablet_service.cc:1468] Processing CreateTablet for tablet 745cd19f3b964febbba91f9bf95755c8 (DEFAULT_TABLE table=default.Table_c2765164e15a4bc8adeae4612f437ce2 [id=0cd6f94f154e4eb3a0855b42717e72b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.249234 1162 consensus_queue.cc:260] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.249789 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb (DEFAULT_TABLE table=default.Table_dbc79667956e45a6963ad166c7a8440d [id=1b9ac6804b0c4c35b721470779b09901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.250686 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb (DEFAULT_TABLE table=default.Table_dbc79667956e45a6963ad166c7a8440d [id=1b9ac6804b0c4c35b721470779b09901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.251647 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 7942f68373e24e9aa23d5ee80707432e (DEFAULT_TABLE table=default.Table_d9f68c7739ec412393bfcc8b1440d7a8 [id=ae2d05b9fa8c4fb8b67574b6b0c53771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.252508 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7942f68373e24e9aa23d5ee80707432e (DEFAULT_TABLE table=default.Table_d9f68c7739ec412393bfcc8b1440d7a8 [id=ae2d05b9fa8c4fb8b67574b6b0c53771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.252903 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7942f68373e24e9aa23d5ee80707432e (DEFAULT_TABLE table=default.Table_d9f68c7739ec412393bfcc8b1440d7a8 [id=ae2d05b9fa8c4fb8b67574b6b0c53771]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.253175 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 (DEFAULT_TABLE table=default.Table_d6623a19ca9c41db9ea86fb93a800ce4 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.253249 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7942f68373e24e9aa23d5ee80707432e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.253621 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:25.254987 881 tablet_service.cc:1468] Processing CreateTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 (DEFAULT_TABLE table=default.Table_d6623a19ca9c41db9ea86fb93a800ce4 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.255764 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e972cfec558b45e287ef91014173b74d (DEFAULT_TABLE table=default.Table_7d746365978745ddb6d89f0e71262bad [id=cbe28ac967424da9bf7ccddd2a66f496]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.192109 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 (DEFAULT_TABLE table=default.Table_2e1e160bec8647f2b41862f84d3e4817 [id=7f48c4b1013f4831a97566298978262b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:25.257105 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8f7240577a874a59b3268ff7f0c1beec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f7240577a874a59b3268ff7f0c1beec already in progress: creating tablet
I20250625 07:52:25.258981 1166 ts_tablet_manager.cc:1905] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.259186 750 tablet_service.cc:1468] Processing CreateTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 (DEFAULT_TABLE table=default.Table_d6623a19ca9c41db9ea86fb93a800ce4 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.260126 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ab63bf0a0f4be29cb027505cedf835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.257524 1161 tablet_replica.cc:331] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:25.269326 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9b46e6d83dcd438fb7a25fb360da7435"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.269968 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b46e6d83dcd438fb7a25fb360da7435 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.271791 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3d1a2916527442ea8dc34a8b834ea1f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.272300 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d1a2916527442ea8dc34a8b834ea1f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.272763 754 tablet_service.cc:1468] Processing CreateTablet for tablet e972cfec558b45e287ef91014173b74d (DEFAULT_TABLE table=default.Table_7d746365978745ddb6d89f0e71262bad [id=cbe28ac967424da9bf7ccddd2a66f496]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.273175 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e972cfec558b45e287ef91014173b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.238381 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef2eb85358b414c9755f81f97beaddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.277060 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_014505d5378f4c5a89fefb3ada354601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.277586 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_014505d5378f4c5a89fefb3ada354601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.277555 877 tablet_service.cc:1468] Processing CreateTablet for tablet e972cfec558b45e287ef91014173b74d (DEFAULT_TABLE table=default.Table_7d746365978745ddb6d89f0e71262bad [id=cbe28ac967424da9bf7ccddd2a66f496]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.279275 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb (DEFAULT_TABLE table=default.Table_dbc79667956e45a6963ad166c7a8440d [id=1b9ac6804b0c4c35b721470779b09901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.279719 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.280836 1162 ts_tablet_manager.cc:1428] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.033s user 0.000s sys 0.001s
I20250625 07:52:25.281363 1162 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:25.281400 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 38b468877c0c412aaa0ff42cffded4c5 (DEFAULT_TABLE table=default.Table_302f6068ad054b38aa3d44e3dd369c8e [id=1ee5d09a5efd4dfb99e4f69f08262f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.282359 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5cbace0f99a44800bab1de5e38eb5112
I20250625 07:52:25.282861 590 catalog_manager.cc:6652] ReplaceTablet: tablet 5cbace0f99a44800bab1de5e38eb5112 deleted and replaced by tablet f5542f74374e4efe9acc6adc2c6e5b46
I20250625 07:52:25.283143 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 with delete_type TABLET_DATA_DELETED (replaced by tablet f5542f74374e4efe9acc6adc2c6e5b46) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.283552 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b620bb670cd94dce999cbe1dabc6125a
I20250625 07:52:25.284024 1162 tablet_bootstrap.cc:654] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.284708 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b620bb670cd94dce999cbe1dabc6125a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.285323 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 with delete_type TABLET_DATA_DELETED (replaced by tablet f5542f74374e4efe9acc6adc2c6e5b46) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.285574 749 tablet_service.cc:1515] Processing DeleteTablet for tablet b620bb670cd94dce999cbe1dabc6125a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.283860 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 5cbace0f99a44800bab1de5e38eb5112 with delete_type TABLET_DATA_DELETED (replaced by tablet f5542f74374e4efe9acc6adc2c6e5b46) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.284374 880 tablet_service.cc:1515] Processing DeleteTablet for tablet b620bb670cd94dce999cbe1dabc6125a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.287506 1729 raft_consensus.cc:491] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.289907 1729 raft_consensus.cc:513] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.288486 749 tablet_service.cc:1468] Processing CreateTablet for tablet 38b468877c0c412aaa0ff42cffded4c5 (DEFAULT_TABLE table=default.Table_302f6068ad054b38aa3d44e3dd369c8e [id=1ee5d09a5efd4dfb99e4f69f08262f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.291175 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b468877c0c412aaa0ff42cffded4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.291549 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "450727d8f33e4eb489969b7974cdeeb1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:25.292302 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "450727d8f33e4eb489969b7974cdeeb1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.292488 850 leader_election.cc:343] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:25.292984 851 leader_election.cc:343] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.293100 851 leader_election.cc:304] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.293352 1766 raft_consensus.cc:2747] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.293563 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7d746365978745ddb6d89f0e71262bad" } new_table_name: "default.Table_3a3e42d99e2541b59a00c5457c457b4f" modify_external_catalogs: true
I20250625 07:52:25.290565 1729 leader_election.cc:290] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.295521 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_44af03fefc224c18ac4dc64986b4fa3d" } new_table_name: "default.Table_e127e5950e034aed9b6b4ec3f6e1af72" modify_external_catalogs: true
W20250625 07:52:25.297518 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02ec8c9e6340497abe802c7345f42921 consensus state has no leader
I20250625 07:52:25.303725 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b8f8397cf384480e8f65fcbf6cd5de9d" } new_table_name: "default.Table_a23788a3c209434db9f0ef44e0220672" modify_external_catalogs: true
I20250625 07:52:25.304598 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ab63bf0a0f4be29cb027505cedf835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.304826 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac8968da295c4b7fb6d83b84935d27d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.305325 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7631c038cdc447ce8e361f857334153b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.306463 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7631c038cdc447ce8e361f857334153b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.308077 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b9c59f89eb80476cb447870b7c2b4251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.308634 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9c59f89eb80476cb447870b7c2b4251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.308835 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b294489423ab06fa7c6988b75f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.310528 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b468877c0c412aaa0ff42cffded4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.310647 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c3a4e707895c4f80995d1f26e93afe45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.311144 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3a4e707895c4f80995d1f26e93afe45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.311612 880 tablet_service.cc:1468] Processing CreateTablet for tablet 38b468877c0c412aaa0ff42cffded4c5 (DEFAULT_TABLE table=default.Table_302f6068ad054b38aa3d44e3dd369c8e [id=1ee5d09a5efd4dfb99e4f69f08262f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.312610 892 tablet_service.cc:1468] Processing CreateTablet for tablet b6661b294489423ab06fa7c6988b75f7 (DEFAULT_TABLE table=default.Table_a4a61d6fd2514d88989f611b35c12baa [id=f07bd577df8545a7a31a2300e2662d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.312840 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.314417 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:25.316812 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a0fe49e81af4d8eb81271cf7b8057d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.317643 1024 tablet_service.cc:1468] Processing CreateTablet for tablet a25f89bffcd0430a976113d512c6a699 (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.317960 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25f89bffcd0430a976113d512c6a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.318393 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e972cfec558b45e287ef91014173b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.320801 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca2f298aefc42e890d0ae48f26ba1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.323840 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745cd19f3b964febbba91f9bf95755c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.325053 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799be9c1ae93465d9f70fc960d50bec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.329134 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.330435 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6661b294489423ab06fa7c6988b75f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.333513 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745cd19f3b964febbba91f9bf95755c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.333582 1733 raft_consensus.cc:491] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.333688 1733 raft_consensus.cc:513] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.334295 1733 leader_election.cc:290] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:25.334702 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.334841 713 consensus_peers.cc:487] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:25.335214 915 raft_consensus.cc:1273] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:25.335965 1660 consensus_queue.cc:1035] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:25.336499 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.336946 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7942f68373e24e9aa23d5ee80707432e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.340859 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4914a64e704a4386b8241e95229c3786" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:25.341003 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b927ec564efa4ff080d22a673abb9b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.341514 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b468877c0c412aaa0ff42cffded4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.344969 985 leader_election.cc:304] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:25.345085 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7942f68373e24e9aa23d5ee80707432e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.345386 1733 raft_consensus.cc:2747] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.348954 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ab63bf0a0f4be29cb027505cedf835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.352948 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e972cfec558b45e287ef91014173b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.281710 1163 ts_tablet_manager.cc:1428] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.041s user 0.002s sys 0.004s
I20250625 07:52:25.372812 1161 raft_consensus.cc:2241] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.373015 1161 raft_consensus.cc:2270] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.373970 1163 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.377046 1163 tablet_bootstrap.cc:654] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.380818 1161 ts_tablet_manager.cc:1905] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.382498 1660 raft_consensus.cc:491] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.382681 1660 raft_consensus.cc:513] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.383258 1660 leader_election.cc:290] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:25.383975 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:25.384157 912 raft_consensus.cc:2466] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:25.384105 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.384665 713 leader_election.cc:343] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.385335 1162 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:25.385468 1162 ts_tablet_manager.cc:1397] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.022s
I20250625 07:52:25.385815 713 leader_election.cc:304] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.386103 1660 raft_consensus.cc:2802] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:25.386204 1660 raft_consensus.cc:491] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:25.386276 1660 raft_consensus.cc:3058] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:25.389338 1162 raft_consensus.cc:357] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.389521 1162 raft_consensus.cc:383] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.389578 1162 raft_consensus.cc:738] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.389847 1162 consensus_queue.cc:260] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.391252 895 tablet_service.cc:1468] Processing CreateTablet for tablet a25f89bffcd0430a976113d512c6a699 (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.392464 749 tablet_service.cc:1468] Processing CreateTablet for tablet a25f89bffcd0430a976113d512c6a699 (DEFAULT_TABLE table=default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.393101 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25f89bffcd0430a976113d512c6a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.397601 1166 ts_tablet_manager.cc:1918] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:25.397706 1166 log.cc:1199] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0e4519b1fbf34ca8b33655f4d402aafc
I20250625 07:52:25.398008 1166 ts_tablet_manager.cc:1939] T 0e4519b1fbf34ca8b33655f4d402aafc P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.398345 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25f89bffcd0430a976113d512c6a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.399163 1163 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.399302 1163 ts_tablet_manager.cc:1397] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250625 07:52:25.399698 1660 raft_consensus.cc:513] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.399937 1163 raft_consensus.cc:357] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.400085 1163 raft_consensus.cc:383] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.400146 1163 raft_consensus.cc:738] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.400295 1660 leader_election.cc:290] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:25.400332 1163 consensus_queue.cc:260] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.400796 1163 ts_tablet_manager.cc:1428] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 07:52:25.400985 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:25.401463 1163 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.401535 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:25.401686 912 raft_consensus.cc:3058] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:25.401736 713 leader_election.cc:343] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.405694 1163 tablet_bootstrap.cc:654] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.405987 1162 ts_tablet_manager.cc:1428] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250625 07:52:25.410377 1162 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:25.412458 1162 tablet_bootstrap.cc:654] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.417670 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9f09205826714483b879b30862cdf1c9" } modify_external_catalogs: true
I20250625 07:52:25.417930 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9f09205826714483b879b30862cdf1c9" } modify_external_catalogs: true
I20250625 07:52:25.422341 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d294d1a1c5f749e9ae13e0bb5e4f404e" } new_table_name: "default.Table_d218655737a6439da5c24f4643f48c76" modify_external_catalogs: true
I20250625 07:52:25.431663 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d86f89948c7043899036f6af6183d950"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.432442 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86f89948c7043899036f6af6183d950 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.434062 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bcddc07c3a8a46449f20441de84992f6" } modify_external_catalogs: true
I20250625 07:52:25.434350 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bcddc07c3a8a46449f20441de84992f6" } modify_external_catalogs: true
W20250625 07:52:25.436100 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7f82e83ed924d2bbedd33d96642a10a consensus state has no leader
I20250625 07:52:25.436591 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f35dcb866b4e4dcc98712f798468a51e" } modify_external_catalogs: true
I20250625 07:52:25.436803 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f35dcb866b4e4dcc98712f798468a51e" } modify_external_catalogs: true
W20250625 07:52:25.439133 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e972cfec558b45e287ef91014173b74d consensus state has no leader
I20250625 07:52:25.440802 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_913332b4f3224e12941ecc4c9d33f0d2" } modify_external_catalogs: true
I20250625 07:52:25.441051 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_913332b4f3224e12941ecc4c9d33f0d2" } modify_external_catalogs: true
W20250625 07:52:25.443022 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abb4d9dc2da7453d8336d5589dda3a74 consensus state has no leader
I20250625 07:52:25.444370 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.448717 1163 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.448873 1163 ts_tablet_manager.cc:1397] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20250625 07:52:25.449950 1163 raft_consensus.cc:357] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.450129 1163 raft_consensus.cc:383] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.450186 1163 raft_consensus.cc:738] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.450369 1163 consensus_queue.cc:260] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.450868 1163 ts_tablet_manager.cc:1428] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:25.452829 1163 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.454380 1163 tablet_bootstrap.cc:654] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.457259 1760 tablet.cc:1722] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:25.463048 1761 tablet.cc:1722] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:25.465114 1158 tablet_bootstrap.cc:492] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:25.465312 1158 ts_tablet_manager.cc:1397] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.303s user 0.003s sys 0.001s
I20250625 07:52:25.466104 1158 raft_consensus.cc:357] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.466323 1158 raft_consensus.cc:383] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.466411 1158 raft_consensus.cc:738] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.466662 1158 consensus_queue.cc:260] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } }
I20250625 07:52:25.467173 1158 ts_tablet_manager.cc:1428] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:25.468937 1158 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:25.469959 1163 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.470091 1163 ts_tablet_manager.cc:1397] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20250625 07:52:25.470748 1163 raft_consensus.cc:357] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.470897 1163 raft_consensus.cc:383] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.470940 1163 raft_consensus.cc:738] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.471160 1163 consensus_queue.cc:260] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.471581 1163 ts_tablet_manager.cc:1428] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:25.471807 1163 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.473284 1163 tablet_bootstrap.cc:654] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.478016 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:25.480929 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0e4519b1fbf34ca8b33655f4d402aafc (table default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]) successfully deleted
W20250625 07:52:25.481194 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4f3b6dc279984d7fba83a6308fbc0fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3b6dc279984d7fba83a6308fbc0fba already in progress: creating tablet
W20250625 07:52:25.481426 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5cbace0f99a44800bab1de5e38eb5112 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbace0f99a44800bab1de5e38eb5112 already in progress: creating tablet
W20250625 07:52:25.481621 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b620bb670cd94dce999cbe1dabc6125a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b620bb670cd94dce999cbe1dabc6125a already in progress: creating tablet
I20250625 07:52:25.486174 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.488426 1163 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.488557 1163 ts_tablet_manager.cc:1397] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250625 07:52:25.489414 1163 raft_consensus.cc:357] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.489576 1163 raft_consensus.cc:383] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.489620 1163 raft_consensus.cc:738] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.489782 1163 consensus_queue.cc:260] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.490195 1163 ts_tablet_manager.cc:1428] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:25.490440 1163 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.492432 1163 tablet_bootstrap.cc:654] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.496445 912 raft_consensus.cc:2466] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:25.496587 1161 ts_tablet_manager.cc:1918] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.496717 1161 log.cc:1199] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0452e73865964f709310fd22ac39a5f0
I20250625 07:52:25.496999 1161 ts_tablet_manager.cc:1939] T 0452e73865964f709310fd22ac39a5f0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:25.500232 713 leader_election.cc:304] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.501035 1543 raft_consensus.cc:2802] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:25.501222 1543 raft_consensus.cc:695] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:25.501448 1543 consensus_queue.cc:237] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.502460 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet e76df9405d91488dad0cfa7f8755bb1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:12 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.502696 1161 tablet_replica.cc:331] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:25.502846 1161 raft_consensus.cc:2241] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.502974 1161 raft_consensus.cc:2270] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.505702 1163 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.505841 1163 ts_tablet_manager.cc:1397] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20250625 07:52:25.506682 1163 raft_consensus.cc:357] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.506834 1163 raft_consensus.cc:383] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.506880 1163 raft_consensus.cc:738] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.507077 1163 consensus_queue.cc:260] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.507491 1163 ts_tablet_manager.cc:1428] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:25.507730 1163 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.507788 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0452e73865964f709310fd22ac39a5f0 (table default.Table_c589a7d832eb4e7fb74f00c8fcbaa680 [id=7ce5c0adc972450bb89be40d18e4cc30]) successfully deleted
W20250625 07:52:25.508134 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6a6fab215b63465a9e48738d3f27651e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a6fab215b63465a9e48738d3f27651e already in progress: creating tablet
W20250625 07:52:25.508354 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fd9677bf99f041c2ade65c94f0170609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd9677bf99f041c2ade65c94f0170609 already in progress: creating tablet
W20250625 07:52:25.508577 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4f3b6dc279984d7fba83a6308fbc0fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3b6dc279984d7fba83a6308fbc0fba already in progress: creating tablet
I20250625 07:52:25.509481 1163 tablet_bootstrap.cc:654] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:25.511329 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0e4519b1fbf34ca8b33655f4d402aafc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e4519b1fbf34ca8b33655f4d402aafc already in progress: creating tablet
W20250625 07:52:25.511619 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5cbace0f99a44800bab1de5e38eb5112 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbace0f99a44800bab1de5e38eb5112 already in progress: creating tablet
W20250625 07:52:25.511816 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b620bb670cd94dce999cbe1dabc6125a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b620bb670cd94dce999cbe1dabc6125a already in progress: creating tablet
I20250625 07:52:25.519599 1161 ts_tablet_manager.cc:1905] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.527163 1163 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.527318 1163 ts_tablet_manager.cc:1397] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20250625 07:52:25.528117 1163 raft_consensus.cc:357] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.528294 1163 raft_consensus.cc:383] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.528347 1163 raft_consensus.cc:738] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.528529 1163 consensus_queue.cc:260] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.528990 1163 ts_tablet_manager.cc:1428] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:25.529418 1163 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.530980 1163 tablet_bootstrap.cc:654] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.542503 1163 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.542649 1163 ts_tablet_manager.cc:1397] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20250625 07:52:25.543818 1163 raft_consensus.cc:357] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.543985 1163 raft_consensus.cc:383] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.544032 1163 raft_consensus.cc:738] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.544227 1163 consensus_queue.cc:260] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.544679 1163 ts_tablet_manager.cc:1428] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:25.544958 1163 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.546547 1163 tablet_bootstrap.cc:654] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:25.550559 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fdad7dc1b27e4d87a747642a6d47230c (table default.Table_ba10e2f968a543d4a5173431533d7e72 [id=6da9e5a92547450b83830b27727c69ef], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet fdad7dc1b27e4d87a747642a6d47230c consensus state has no leader
I20250625 07:52:25.565758 1163 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.565918 1163 ts_tablet_manager.cc:1397] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250625 07:52:25.566653 1163 raft_consensus.cc:357] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.566823 1163 raft_consensus.cc:383] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.566874 1163 raft_consensus.cc:738] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.567042 1163 consensus_queue.cc:260] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.567464 1163 ts_tablet_manager.cc:1428] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:25.567713 1163 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.567801 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3cede47fcd084b178dd44942300d7a4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.568431 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cede47fcd084b178dd44942300d7a4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.569758 1163 tablet_bootstrap.cc:654] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.574782 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dc8c4809369446dc9c12a878a16e6c48" } modify_external_catalogs: true
I20250625 07:52:25.575001 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dc8c4809369446dc9c12a878a16e6c48" } modify_external_catalogs: true
W20250625 07:52:25.574810 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eba09cd15d9f4611ab14d4a6dc372719 has no consensus state
I20250625 07:52:25.575655 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f2029eb1ac264cc9967ec89a919d5431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.576192 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2029eb1ac264cc9967ec89a919d5431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.587651 1161 ts_tablet_manager.cc:1918] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.587771 1161 log.cc:1199] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e76df9405d91488dad0cfa7f8755bb1c
I20250625 07:52:25.588034 1161 ts_tablet_manager.cc:1939] T e76df9405d91488dad0cfa7f8755bb1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:25.592278 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8aa30ca0035048d3b60669994d402734
I20250625 07:52:25.593386 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b1cdde72919a4a3ca29434d475c2d44f
I20250625 07:52:25.595469 892 tablet_service.cc:1515] Processing DeleteTablet for tablet b1cdde72919a4a3ca29434d475c2d44f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.595764 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa30ca0035048d3b60669994d402734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.595762 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa30ca0035048d3b60669994d402734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.595938 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet b1cdde72919a4a3ca29434d475c2d44f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.596181 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_26b14eff630948cb807b96dfd6e192c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.596801 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26b14eff630948cb807b96dfd6e192c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.598080 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c2765164e15a4bc8adeae4612f437ce2" } new_table_name: "default.Table_e22e84e3af7b408ebf73a2c0cdb15964" modify_external_catalogs: true
I20250625 07:52:25.598443 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa30ca0035048d3b60669994d402734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.598668 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b1cdde72919a4a3ca29434d475c2d44f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.598733 1166 tablet_replica.cc:331] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.598986 1166 raft_consensus.cc:2241] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:25.599417 1166 raft_consensus.cc:2270] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.600165 1166 ts_tablet_manager.cc:1905] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.600888 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a] alter complete (version 1)
I20250625 07:52:25.600941 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e41e53efb6f342d2b1e4d642f7aaad10
I20250625 07:52:25.605429 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.607142 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.609484 1729 raft_consensus.cc:491] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.609640 1729 raft_consensus.cc:513] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.610319 1729 leader_election.cc:290] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.610838 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db110bb6cc146a393d62ddabbe1955a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.611361 851 leader_election.cc:343] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.611364 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.611938 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.612288 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db110bb6cc146a393d62ddabbe1955a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.612916 850 leader_election.cc:343] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.613040 850 leader_election.cc:304] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.613517 1729 raft_consensus.cc:2747] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.613848 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.615396 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.617000 895 tablet_service.cc:1515] Processing DeleteTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.617854 892 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.618664 892 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.619334 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.625986 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_72eb667adefe48c582a543cd03322971"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.626721 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72eb667adefe48c582a543cd03322971 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.629756 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_997a282eabfa48049916f38d2f6b7089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.630465 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_997a282eabfa48049916f38d2f6b7089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.629755 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f07796775e9f4e0299fa92332951f3f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.632546 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f07796775e9f4e0299fa92332951f3f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.634263 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e76df9405d91488dad0cfa7f8755bb1c (table default.Table_9c43e2a9a4474aed9941962b1623bcaa [id=8714fe465e484aa29ebd16b5677accd2]) successfully deleted
W20250625 07:52:25.636204 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8aa30ca0035048d3b60669994d402734 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa30ca0035048d3b60669994d402734 already in progress: creating tablet
W20250625 07:52:25.636449 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b1cdde72919a4a3ca29434d475c2d44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cdde72919a4a3ca29434d475c2d44f already in progress: creating tablet
W20250625 07:52:25.636675 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e41e53efb6f342d2b1e4d642f7aaad10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e41e53efb6f342d2b1e4d642f7aaad10 already in progress: creating tablet
I20250625 07:52:25.638926 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e35c55dc42af4bb28966deeb87d520c8" } new_table_name: "default.Table_9bc3d032aee948de91f3c5c8c776cf7d" modify_external_catalogs: true
W20250625 07:52:25.640305 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 745cd19f3b964febbba91f9bf95755c8 consensus state has no leader
I20250625 07:52:25.641502 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5117224af8bf44ee866ec4497721cf63" } new_table_name: "default.Table_56a7bd0b475e451b8afa385ff9915ba5" modify_external_catalogs: true
I20250625 07:52:25.644470 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3786a3e21ca46cdb5365fe9bc606a7f" } new_table_name: "default.Table_fd84ef6e2ba44f1ebc48f42fcdbc7d37" modify_external_catalogs: true
I20250625 07:52:25.651378 593 catalog_manager.cc:5582] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:25.651836 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:25.652504 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_8b2d59eebc0448e78ca54efc7537619e [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 1 got 0
I20250625 07:52:25.655696 1021 tablet_service.cc:1468] Processing CreateTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 (DEFAULT_TABLE table=default.Table_d218655737a6439da5c24f4643f48c76 [id=a9537692e261469cbf71ad34591fdbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.656163 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba09cd15d9f4611ab14d4a6dc372719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.657033 892 tablet_service.cc:1468] Processing CreateTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 (DEFAULT_TABLE table=default.Table_d218655737a6439da5c24f4643f48c76 [id=a9537692e261469cbf71ad34591fdbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.657434 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba09cd15d9f4611ab14d4a6dc372719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.659353 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 1da321f1ad34485a92a0c3f4c39aa444 (DEFAULT_TABLE table=default.Table_d29d1b5f858f41d590392cfd6ec5c0d9 [id=55db942c7adf4f6fa17aefb4a781deaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.659725 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da321f1ad34485a92a0c3f4c39aa444. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.660027 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1da321f1ad34485a92a0c3f4c39aa444 (DEFAULT_TABLE table=default.Table_d29d1b5f858f41d590392cfd6ec5c0d9 [id=55db942c7adf4f6fa17aefb4a781deaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.660344 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da321f1ad34485a92a0c3f4c39aa444. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.660943 1012 tablet_service.cc:1468] Processing CreateTablet for tablet f35ac535578d4681ac998a530b1f1d4f (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.661217 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35ac535578d4681ac998a530b1f1d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.662905 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a8d9333ac16747b48830505bb5f1c2d5 (DEFAULT_TABLE table=default.Table_f001237d203b42898ff32eff812b5557 [id=42acc6f5af49448f836b384ec92a7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.663237 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d9333ac16747b48830505bb5f1c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.663604 880 tablet_service.cc:1468] Processing CreateTablet for tablet a8d9333ac16747b48830505bb5f1c2d5 (DEFAULT_TABLE table=default.Table_f001237d203b42898ff32eff812b5557 [id=42acc6f5af49448f836b384ec92a7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.663937 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d9333ac16747b48830505bb5f1c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.665987 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.666298 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5542f74374e4efe9acc6adc2c6e5b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.666939 895 tablet_service.cc:1468] Processing CreateTablet for tablet f35ac535578d4681ac998a530b1f1d4f (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.667495 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35ac535578d4681ac998a530b1f1d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.668941 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9 (DEFAULT_TABLE table=default.Table_cc08960a4b664c009b763304b167cf6b [id=5e78a17cd996478698ae31830fd9fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.669267 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.670853 747 tablet_service.cc:1468] Processing CreateTablet for tablet 1da321f1ad34485a92a0c3f4c39aa444 (DEFAULT_TABLE table=default.Table_d29d1b5f858f41d590392cfd6ec5c0d9 [id=55db942c7adf4f6fa17aefb4a781deaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.671077 750 tablet_service.cc:1468] Processing CreateTablet for tablet a8d9333ac16747b48830505bb5f1c2d5 (DEFAULT_TABLE table=default.Table_f001237d203b42898ff32eff812b5557 [id=42acc6f5af49448f836b384ec92a7d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.671202 744 tablet_service.cc:1468] Processing CreateTablet for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9 (DEFAULT_TABLE table=default.Table_cc08960a4b664c009b763304b167cf6b [id=5e78a17cd996478698ae31830fd9fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.671625 755 tablet_service.cc:1468] Processing CreateTablet for tablet 457faa279d66483686359908026a2b27 (DEFAULT_TABLE table=default.Table_f3fa68b5cb55490083020c843fe1108a [id=2c4e9ea66ba3474f968b19058eca283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.671743 881 tablet_service.cc:1468] Processing CreateTablet for tablet 457faa279d66483686359908026a2b27 (DEFAULT_TABLE table=default.Table_f3fa68b5cb55490083020c843fe1108a [id=2c4e9ea66ba3474f968b19058eca283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.672056 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 457faa279d66483686359908026a2b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.671999 742 tablet_service.cc:1468] Processing CreateTablet for tablet f54fa729d3754d7eb7e0528a733d40b6 (DEFAULT_TABLE table=default.Table_e753ff3879ec40f58838c14425717b90 [id=9f2e39e91a134dcb8fa2c9a0bf97cf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.672386 751 tablet_service.cc:1468] Processing CreateTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.672776 754 tablet_service.cc:1468] Processing CreateTablet for tablet f35ac535578d4681ac998a530b1f1d4f (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.672937 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 457faa279d66483686359908026a2b27 (DEFAULT_TABLE table=default.Table_f3fa68b5cb55490083020c843fe1108a [id=2c4e9ea66ba3474f968b19058eca283e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.673141 743 tablet_service.cc:1468] Processing CreateTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 (DEFAULT_TABLE table=default.Table_d218655737a6439da5c24f4643f48c76 [id=a9537692e261469cbf71ad34591fdbd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.673229 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 457faa279d66483686359908026a2b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.673959 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d9333ac16747b48830505bb5f1c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.674835 877 tablet_service.cc:1468] Processing CreateTablet for tablet f54fa729d3754d7eb7e0528a733d40b6 (DEFAULT_TABLE table=default.Table_e753ff3879ec40f58838c14425717b90 [id=9f2e39e91a134dcb8fa2c9a0bf97cf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.675119 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54fa729d3754d7eb7e0528a733d40b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.676113 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54fa729d3754d7eb7e0528a733d40b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.676529 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 457faa279d66483686359908026a2b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.676602 1013 tablet_service.cc:1468] Processing CreateTablet for tablet f54fa729d3754d7eb7e0528a733d40b6 (DEFAULT_TABLE table=default.Table_e753ff3879ec40f58838c14425717b90 [id=9f2e39e91a134dcb8fa2c9a0bf97cf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.676887 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5542f74374e4efe9acc6adc2c6e5b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.676908 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f54fa729d3754d7eb7e0528a733d40b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.677250 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.677618 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35ac535578d4681ac998a530b1f1d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.677979 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba09cd15d9f4611ab14d4a6dc372719. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.677932 893 tablet_service.cc:1468] Processing CreateTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb (DEFAULT_TABLE table=default.Table_e13b99b01ed747aeaf4c7005e326366a [id=99054a8d18704dc1b7584d5a0068aae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.678843 758 tablet_service.cc:1468] Processing CreateTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb (DEFAULT_TABLE table=default.Table_e13b99b01ed747aeaf4c7005e326366a [id=99054a8d18704dc1b7584d5a0068aae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.679854 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3284c494179742749faa4d4f74ed7b25 (DEFAULT_TABLE table=default.Table_b47b10ad384d42129c9bf8891ecd6673 [id=d1f7dea691f6400487cdb6eb7561d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.680200 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3284c494179742749faa4d4f74ed7b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.680919 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3284c494179742749faa4d4f74ed7b25 (DEFAULT_TABLE table=default.Table_b47b10ad384d42129c9bf8891ecd6673 [id=d1f7dea691f6400487cdb6eb7561d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.680927 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb (DEFAULT_TABLE table=default.Table_e13b99b01ed747aeaf4c7005e326366a [id=99054a8d18704dc1b7584d5a0068aae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.681114 883 tablet_service.cc:1468] Processing CreateTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 (DEFAULT_TABLE table=default.Table_6d71f14d5bd44891a4c2e98f3f43034e [id=cf1dd0fc98824cc3865fdc8ba8d8367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.681223 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fa349c737f4cebb1225b20e62e3dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.681550 885 tablet_service.cc:1468] Processing CreateTablet for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9 (DEFAULT_TABLE table=default.Table_cc08960a4b664c009b763304b167cf6b [id=5e78a17cd996478698ae31830fd9fdd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.681869 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3284c494179742749faa4d4f74ed7b25 (DEFAULT_TABLE table=default.Table_b47b10ad384d42129c9bf8891ecd6673 [id=d1f7dea691f6400487cdb6eb7561d250]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.682343 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fa349c737f4cebb1225b20e62e3dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.682564 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3284c494179742749faa4d4f74ed7b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.682749 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120e64ce7fd94ecf8f9c1b097c60c6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.685771 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 280cd243d39343e8a36ffb8fa541b9b3 (DEFAULT_TABLE table=default.Table_d45fec10075f4eb898fd3cb99709c2ec [id=b8afd5bebb934d3eb8112ea0f701bfa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.686110 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280cd243d39343e8a36ffb8fa541b9b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:25.686488 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f4359fd842a4760bae75169e8914f1c consensus state has no leader
W20250625 07:52:25.686880 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e529cb070cbf4272b900484ce268b291 consensus state has no leader
W20250625 07:52:25.687870 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e46f27344b848318dcd1e782819b75e consensus state has no leader
I20250625 07:52:25.687971 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5542f74374e4efe9acc6adc2c6e5b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.693313 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2afbfb6fdb164ae2917ebf1bbb6433fe
I20250625 07:52:25.694069 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6b39c48d9948400a8f36e389cff0020a
I20250625 07:52:25.693997 753 tablet_service.cc:1468] Processing CreateTablet for tablet 280cd243d39343e8a36ffb8fa541b9b3 (DEFAULT_TABLE table=default.Table_d45fec10075f4eb898fd3cb99709c2ec [id=b8afd5bebb934d3eb8112ea0f701bfa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.694105 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.694696 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b39c48d9948400a8f36e389cff0020a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.695362 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da321f1ad34485a92a0c3f4c39aa444. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.701076 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cebbbf28857c46a6adc44443c9a09488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.701714 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cebbbf28857c46a6adc44443c9a09488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.698207 1166 ts_tablet_manager.cc:1918] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:25.709654 1166 log.cc:1199] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/8aa30ca0035048d3b60669994d402734
I20250625 07:52:25.710059 1166 ts_tablet_manager.cc:1939] T 8aa30ca0035048d3b60669994d402734 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.713104 1166 tablet_replica.cc:331] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.713501 1166 raft_consensus.cc:2241] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.713737 1166 raft_consensus.cc:2270] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.714912 1166 ts_tablet_manager.cc:1905] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.717355 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 8aa30ca0035048d3b60669994d402734 (table default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]) successfully deleted
W20250625 07:52:25.717753 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b1cdde72919a4a3ca29434d475c2d44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cdde72919a4a3ca29434d475c2d44f already in progress: creating tablet
I20250625 07:52:25.720948 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8fa349c737f4cebb1225b20e62e3dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.723233 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280cd243d39343e8a36ffb8fa541b9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.723706 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3284c494179742749faa4d4f74ed7b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.726274 742 tablet_service.cc:1468] Processing CreateTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef (DEFAULT_TABLE table=default.Table_6e438577c44d4924bbfad6fa2cc3beaa [id=59d8a1322acd46949db809bb7b5af759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.726743 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bf5c84b12d4bb399c16eb6e77a28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.734432 747 tablet_service.cc:1468] Processing CreateTablet for tablet f396da8e4b484a92bed4c1893d4e9e39 (DEFAULT_TABLE table=default.Table_d841dee5b83d4242b8644949082fa7a4 [id=a1c7cc4ebcac493f85a1bd3f9f639584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.734826 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396da8e4b484a92bed4c1893d4e9e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.736829 1158 tablet_bootstrap.cc:700] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent opening tablet: real 0.266s user 0.000s sys 0.000s
I20250625 07:52:25.737200 1158 tablet_bootstrap.cc:654] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.746573 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e with delete_type TABLET_DATA_DELETED (replaced by tablet a25f89bffcd0430a976113d512c6a699) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.748293 749 tablet_service.cc:1468] Processing CreateTablet for tablet a78179b5e72b40bbacd7a707839cb152 (DEFAULT_TABLE table=default.Table_5b6911bd0ce04836b8e4215c59c5847e [id=8be576f29b19427bbbe0504551d9dd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.749496 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78179b5e72b40bbacd7a707839cb152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.753758 1166 ts_tablet_manager.cc:1918] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:25.758536 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.759346 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.759660 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b39c48d9948400a8f36e389cff0020a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.759403 1160 ts_tablet_manager.cc:1918] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.759872 1160 log.cc:1199] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:25.760063 1158 tablet_bootstrap.cc:492] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:25.760161 1160 ts_tablet_manager.cc:1939] T 366d3e6fcc9d49be8fbd2cd070632861 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.760215 1158 ts_tablet_manager.cc:1397] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.291s user 0.003s sys 0.001s
I20250625 07:52:25.760679 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b39c48d9948400a8f36e389cff0020a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.761138 892 tablet_service.cc:1468] Processing CreateTablet for tablet 280cd243d39343e8a36ffb8fa541b9b3 (DEFAULT_TABLE table=default.Table_d45fec10075f4eb898fd3cb99709c2ec [id=b8afd5bebb934d3eb8112ea0f701bfa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.761355 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f396da8e4b484a92bed4c1893d4e9e39 (DEFAULT_TABLE table=default.Table_d841dee5b83d4242b8644949082fa7a4 [id=a1c7cc4ebcac493f85a1bd3f9f639584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.761548 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 280cd243d39343e8a36ffb8fa541b9b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.761708 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396da8e4b484a92bed4c1893d4e9e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.766727 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b7f6776839094087962a6b3d39f30ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.767346 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f6776839094087962a6b3d39f30ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.770325 877 tablet_service.cc:1468] Processing CreateTablet for tablet f396da8e4b484a92bed4c1893d4e9e39 (DEFAULT_TABLE table=default.Table_d841dee5b83d4242b8644949082fa7a4 [id=a1c7cc4ebcac493f85a1bd3f9f639584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.770927 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f396da8e4b484a92bed4c1893d4e9e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.771149 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef (DEFAULT_TABLE table=default.Table_6e438577c44d4924bbfad6fa2cc3beaa [id=59d8a1322acd46949db809bb7b5af759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.771512 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bf5c84b12d4bb399c16eb6e77a28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.774940 1166 log.cc:1199] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e41e53efb6f342d2b1e4d642f7aaad10
I20250625 07:52:25.775379 1166 ts_tablet_manager.cc:1939] T e41e53efb6f342d2b1e4d642f7aaad10 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.776897 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9f083e3394b84fcdae11d916769e9f57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.777339 1166 tablet_replica.cc:331] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.777571 1166 raft_consensus.cc:2241] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.777740 1166 raft_consensus.cc:2270] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:25.777783 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f083e3394b84fcdae11d916769e9f57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:25.778378 1788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:25.779524 1166 ts_tablet_manager.cc:1905] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.779867 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8ea66a5cc6ee4a86af88fa5a5d6fd379" } new_table_name: "default.Table_616a034f739349138890dc31bd1d19c1" modify_external_catalogs: true
I20250625 07:52:25.780412 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]) successfully deleted
W20250625 07:52:25.780668 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2afbfb6fdb164ae2917ebf1bbb6433fe already in progress: creating tablet
W20250625 07:52:25.780926 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6b39c48d9948400a8f36e389cff0020a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b39c48d9948400a8f36e389cff0020a already in progress: creating tablet
I20250625 07:52:25.789487 1048 raft_consensus.cc:3058] T 4914a64e704a4386b8241e95229c3786 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:25.796547 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73ec635d7d5242e8804dd3527457a540 consensus state has no leader
I20250625 07:52:25.798280 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6f19a0a49fd04cffa91026e8f47a8bc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.798897 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f19a0a49fd04cffa91026e8f47a8bc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.798899 1166 ts_tablet_manager.cc:1918] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.799026 1166 log.cc:1199] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/08bb47266e8b4a0fa5ec613560b8549e
I20250625 07:52:25.799896 1163 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:25.799981 1011 tablet_service.cc:1468] Processing CreateTablet for tablet a78179b5e72b40bbacd7a707839cb152 (DEFAULT_TABLE table=default.Table_5b6911bd0ce04836b8e4215c59c5847e [id=8be576f29b19427bbbe0504551d9dd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.800370 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78179b5e72b40bbacd7a707839cb152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.802611 1160 tablet_replica.cc:331] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.804134 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 77a127ebc7a74274b60cdd7d08ce22cb from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:25.804525 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1225d003be2b4d048266736844a78a01" } modify_external_catalogs: true
I20250625 07:52:25.804723 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1225d003be2b4d048266736844a78a01" } modify_external_catalogs: true
I20250625 07:52:25.805169 1163 ts_tablet_manager.cc:1397] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.238s user 0.035s sys 0.094s
I20250625 07:52:25.805450 1729 raft_consensus.cc:491] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.805473 1160 raft_consensus.cc:2241] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.805611 1160 raft_consensus.cc:2270] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.805562 1729 raft_consensus.cc:513] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.806151 1160 ts_tablet_manager.cc:1905] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.806329 1163 raft_consensus.cc:357] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.806484 1163 raft_consensus.cc:383] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.806535 1163 raft_consensus.cc:738] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.806669 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f566cc9cdd447798a5cb408ef9cd35a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:25.806694 1163 consensus_queue.cc:260] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:25.807299 851 leader_election.cc:343] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.805359 883 tablet_service.cc:1468] Processing CreateTablet for tablet a78179b5e72b40bbacd7a707839cb152 (DEFAULT_TABLE table=default.Table_5b6911bd0ce04836b8e4215c59c5847e [id=8be576f29b19427bbbe0504551d9dd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.807868 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78179b5e72b40bbacd7a707839cb152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.806169 1729 leader_election.cc:290] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.809854 1729 raft_consensus.cc:491] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.809957 1729 raft_consensus.cc:513] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.810607 1729 leader_election.cc:290] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.817374 1166 ts_tablet_manager.cc:1939] T 08bb47266e8b4a0fa5ec613560b8549e P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.818668 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 08bb47266e8b4a0fa5ec613560b8549e (table default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]) successfully deleted
I20250625 07:52:25.819787 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "743123ffcf8240dc82bea7178559b7ea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.820710 851 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.822641 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_43f007c4611b415b8cdc3a9b8ab76966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.823458 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f007c4611b415b8cdc3a9b8ab76966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.824395 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "743123ffcf8240dc82bea7178559b7ea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.824867 850 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.824959 850 leader_election.cc:304] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.825383 1792 raft_consensus.cc:2747] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.825462 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f566cc9cdd447798a5cb408ef9cd35a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:25.826018 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f8c817fc80f64e76a3078837e90e97ce" } modify_external_catalogs: true
W20250625 07:52:25.826206 850 leader_election.cc:343] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.826277 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f8c817fc80f64e76a3078837e90e97ce" } modify_external_catalogs: true
I20250625 07:52:25.826320 850 leader_election.cc:304] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.826720 1792 raft_consensus.cc:2747] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.828619 1160 ts_tablet_manager.cc:1918] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.829461 1160 log.cc:1199] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:25.830053 1163 ts_tablet_manager.cc:1428] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.025s user 0.009s sys 0.013s
I20250625 07:52:25.830374 1163 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:25.830688 1160 ts_tablet_manager.cc:1939] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.832077 1163 tablet_bootstrap.cc:654] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.837924 1160 tablet_replica.cc:331] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.838364 1160 raft_consensus.cc:2241] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.839030 1160 raft_consensus.cc:2270] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.839792 1160 ts_tablet_manager.cc:1905] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.840380 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_defbe4d5e7684ec5ac3c36f8d600909c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.841285 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_defbe4d5e7684ec5ac3c36f8d600909c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.844664 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f6a3ec7a88e04f038252d1ceb4cdf7a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.845574 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6a3ec7a88e04f038252d1ceb4cdf7a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.848130 885 tablet_service.cc:1468] Processing CreateTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef (DEFAULT_TABLE table=default.Table_6e438577c44d4924bbfad6fa2cc3beaa [id=59d8a1322acd46949db809bb7b5af759]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.848722 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bf5c84b12d4bb399c16eb6e77a28ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.851424 1160 ts_tablet_manager.cc:1918] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:25.854385 742 tablet_service.cc:1468] Processing CreateTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b (DEFAULT_TABLE table=default.Table_72eb667adefe48c582a543cd03322971 [id=1d231960767e497ba040a869d87c47d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.854534 1160 log.cc:1199] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0e4519b1fbf34ca8b33655f4d402aafc
I20250625 07:52:25.854821 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a653e84823d467f9beb897f2bfb3b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.854895 1160 ts_tablet_manager.cc:1939] T 0e4519b1fbf34ca8b33655f4d402aafc P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.855369 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3a3e42d99e2541b59a00c5457c457b4f" } new_table_name: "default.Table_48ce5f2a89b14013b3fcb44470c38912" modify_external_catalogs: true
I20250625 07:52:25.857003 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_be7ed090191c4152b6e6bf3651b6988b" } modify_external_catalogs: true
I20250625 07:52:25.857203 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_be7ed090191c4152b6e6bf3651b6988b" } modify_external_catalogs: true
I20250625 07:52:25.858085 1160 tablet_replica.cc:331] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.858275 1160 raft_consensus.cc:2241] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.858420 1160 raft_consensus.cc:2270] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.859124 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4c1dbe5c89ac43e59af58d81b811f786"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.859644 1160 ts_tablet_manager.cc:1905] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:25.859783 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c1dbe5c89ac43e59af58d81b811f786 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.861502 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5614a7608ac14d9dbf64878ba87a66de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.862020 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5614a7608ac14d9dbf64878ba87a66de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.862277 746 tablet_service.cc:1468] Processing CreateTablet for tablet 33441054063e47deaba2ad04ae6ae764 (DEFAULT_TABLE table=default.Table_f07796775e9f4e0299fa92332951f3f5 [id=b442bda0b9684e9c8daca6f8c82e84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.862672 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33441054063e47deaba2ad04ae6ae764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.862913 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2398a407b3154334a6a167946cf99ad7 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.863317 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398a407b3154334a6a167946cf99ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.864023 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c776bde0fb9d4cdb9ed4ae90f85b078b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.864492 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c776bde0fb9d4cdb9ed4ae90f85b078b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.866886 754 tablet_service.cc:1468] Processing CreateTablet for tablet ccd929ffd026433c9abd3851bb8e1004 (DEFAULT_TABLE table=default.Table_f2029eb1ac264cc9967ec89a919d5431 [id=ca80dfc499bd48d3a56fcc8c8ce4cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.867220 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd929ffd026433c9abd3851bb8e1004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.869024 753 tablet_service.cc:1468] Processing CreateTablet for tablet b7c5c675d0a7470e95a30a499ad15714 (DEFAULT_TABLE table=default.Table_014505d5378f4c5a89fefb3ada354601 [id=413c81222dbb445d9e5275e1e0797d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.869555 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c5c675d0a7470e95a30a499ad15714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.871037 1160 ts_tablet_manager.cc:1918] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.871152 1160 log.cc:1199] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/450727d8f33e4eb489969b7974cdeeb1
I20250625 07:52:25.871420 1160 ts_tablet_manager.cc:1939] T 450727d8f33e4eb489969b7974cdeeb1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.872943 758 tablet_service.cc:1468] Processing CreateTablet for tablet 9f30a8b59beb4311aca534ed322aff03 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.862603 755 tablet_service.cc:1468] Processing CreateTablet for tablet 1742634837e944ffb9cab3646b933950 (DEFAULT_TABLE table=default.Table_997a282eabfa48049916f38d2f6b7089 [id=d400a52f8fde4cddbf984d4eed3823de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.873240 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f30a8b59beb4311aca534ed322aff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.873504 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1742634837e944ffb9cab3646b933950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.875735 1160 tablet_replica.cc:331] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.875911 1160 raft_consensus.cc:2241] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.876045 1160 raft_consensus.cc:2270] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.881510 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0139149bbf574e13be3b0411a83a0593 (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.881865 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0139149bbf574e13be3b0411a83a0593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.884742 1793 raft_consensus.cc:491] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.885485 1793 raft_consensus.cc:513] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.886195 1793 leader_election.cc:290] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.886526 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbdbd9230210455ba615d51f1a98267b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.887099 851 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.887228 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbdbd9230210455ba615d51f1a98267b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.887742 850 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.887836 850 leader_election.cc:304] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.888397 1793 raft_consensus.cc:2747] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.888902 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9aed63ffa6e74306b79919180fb2f25f" } new_table_name: "default.Table_8d5544c1432c4978ada968f7fcbfab59" modify_external_catalogs: true
I20250625 07:52:25.890297 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 77a127ebc7a74274b60cdd7d08ce22cb
I20250625 07:52:25.890807 596 catalog_manager.cc:6652] ReplaceTablet: tablet 77a127ebc7a74274b60cdd7d08ce22cb deleted and replaced by tablet 29a2f1c24e0240c0b1f0007961ee3ad2
I20250625 07:52:25.891822 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 47b4112b7e674518bcc1ea748370fd85
I20250625 07:52:25.891822 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb with delete_type TABLET_DATA_DELETED (replaced by tablet 29a2f1c24e0240c0b1f0007961ee3ad2) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:25.892544 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 77a127ebc7a74274b60cdd7d08ce22cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77a127ebc7a74274b60cdd7d08ce22cb already in progress: creating tablet
I20250625 07:52:25.892660 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 47b4112b7e674518bcc1ea748370fd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.893280 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a0c38fff8ede45cd8909ea2002e45c37" } new_table_name: "default.Table_c5cad6b9d64740e28fa70ff3c5e44001" modify_external_catalogs: true
I20250625 07:52:25.894773 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_58ba6295501d43f4b2c150eb6ac31aa2" } new_table_name: "default.Table_2ad23fe585834c1aba8313557a4fa011" modify_external_catalogs: true
W20250625 07:52:25.895990 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 47b4112b7e674518bcc1ea748370fd85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47b4112b7e674518bcc1ea748370fd85 already in progress: creating tablet
I20250625 07:52:25.897094 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3d40c073684a48a69c3c60f46aab48bc" } modify_external_catalogs: true
W20250625 07:52:25.897451 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e972cfec558b45e287ef91014173b74d consensus state has no leader
I20250625 07:52:25.897619 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3d40c073684a48a69c3c60f46aab48bc" } modify_external_catalogs: true
I20250625 07:52:25.897770 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1444a3af50a54dfe9739e5c255dc27a4" } modify_external_catalogs: true
I20250625 07:52:25.897935 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1444a3af50a54dfe9739e5c255dc27a4" } modify_external_catalogs: true
I20250625 07:52:25.900234 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cac7d20327e14f90af25acb58d724b68
I20250625 07:52:25.901114 746 tablet_service.cc:1515] Processing DeleteTablet for tablet cac7d20327e14f90af25acb58d724b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.902830 747 tablet_service.cc:1468] Processing CreateTablet for tablet 768c9551d33346718af2670a01a626b0 (DEFAULT_TABLE table=default.Table_d86f89948c7043899036f6af6183d950 [id=f3a2c56d0fda43a5893c1992e177c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.903285 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 768c9551d33346718af2670a01a626b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:25.903781 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet cac7d20327e14f90af25acb58d724b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7d20327e14f90af25acb58d724b68 already in progress: creating tablet
I20250625 07:52:25.905015 746 tablet_service.cc:1468] Processing CreateTablet for tablet f5ab0400081848dd872a429685e3b8e2 (DEFAULT_TABLE table=default.Table_cebbbf28857c46a6adc44443c9a09488 [id=de815b21538b4d40a63a02ab538662bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.905381 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ab0400081848dd872a429685e3b8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.905897 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0aeb7077591f466eb77e3096f6b1d74b
W20250625 07:52:25.906306 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a29a9ba27ab4eb494e1b91e4b33aa75 consensus state has no leader
W20250625 07:52:25.906842 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daf0c3772e7941a0a684a29ce160a851 consensus state has no leader
I20250625 07:52:25.907750 1158 raft_consensus.cc:357] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.908000 1158 raft_consensus.cc:383] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.908133 1158 raft_consensus.cc:738] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.908421 1158 consensus_queue.cc:260] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.909945 1158 ts_tablet_manager.cc:1428] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.150s user 0.003s sys 0.000s
I20250625 07:52:25.910430 1158 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:25.912681 1158 tablet_bootstrap.cc:654] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.913610 1162 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:25.914268 1160 ts_tablet_manager.cc:1905] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.914515 1162 ts_tablet_manager.cc:1397] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.504s user 0.030s sys 0.054s
I20250625 07:52:25.915294 1162 raft_consensus.cc:357] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.917691 1162 raft_consensus.cc:383] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.917805 1162 raft_consensus.cc:738] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.918030 1162 consensus_queue.cc:260] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:25.919582 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 128010us
I20250625 07:52:25.919391 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.919858 742 tablet_service.cc:1468] Processing CreateTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 (DEFAULT_TABLE table=default.Table_c3a4e707895c4f80995d1f26e93afe45 [id=bffcae3d59bb4177ae9048be96e9fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.920143 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7965eb5269ad44e0835c74d19fe23861 (DEFAULT_TABLE table=default.Table_3cede47fcd084b178dd44942300d7a4b [id=d1c10ec1fb964fc7bbd6fcf70aef6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.920557 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a91007503ad43cc8f6fae30e4206df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.920732 1793 raft_consensus.cc:491] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.920888 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7965eb5269ad44e0835c74d19fe23861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.920871 1793 raft_consensus.cc:513] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.921471 1793 leader_election.cc:290] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:25.921821 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "494cacf2dcbb40d3b2c9f05f64809870" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:25.922371 851 leader_election.cc:343] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:25.926527 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 127204us
W20250625 07:52:25.927913 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2afbfb6fdb164ae2917ebf1bbb6433fe already in progress: creating tablet
W20250625 07:52:25.928428 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6b39c48d9948400a8f36e389cff0020a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b39c48d9948400a8f36e389cff0020a already in progress: creating tablet
I20250625 07:52:25.929839 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 366d3e6fcc9d49be8fbd2cd070632861 (table default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]) successfully deleted
I20250625 07:52:25.930052 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
W20250625 07:52:25.930264 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 70094e6c7e644f62bf5ef04dcfb4c380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70094e6c7e644f62bf5ef04dcfb4c380 already in progress: creating tablet
W20250625 07:52:25.930491 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 26db0effa17b4cacb1450a01f9631ddf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26db0effa17b4cacb1450a01f9631ddf already in progress: creating tablet
I20250625 07:52:25.930563 1162 ts_tablet_manager.cc:1428] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
W20250625 07:52:25.930708 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0657163b9a0d4f7f99f57f7fd1f32462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0657163b9a0d4f7f99f57f7fd1f32462 already in progress: creating tablet
W20250625 07:52:25.930894 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8f7240577a874a59b3268ff7f0c1beec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f7240577a874a59b3268ff7f0c1beec already in progress: creating tablet
W20250625 07:52:25.931109 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6a6fab215b63465a9e48738d3f27651e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a6fab215b63465a9e48738d3f27651e already in progress: creating tablet
W20250625 07:52:25.931304 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4f3b6dc279984d7fba83a6308fbc0fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f3b6dc279984d7fba83a6308fbc0fba already in progress: creating tablet
W20250625 07:52:25.931499 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fd9677bf99f041c2ade65c94f0170609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd9677bf99f041c2ade65c94f0170609 already in progress: creating tablet
I20250625 07:52:25.931610 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.931677 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0e4519b1fbf34ca8b33655f4d402aafc (table default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]) successfully deleted
I20250625 07:52:25.931849 1166 tablet_replica.cc:331] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.932016 1166 raft_consensus.cc:2241] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.932155 1166 raft_consensus.cc:2270] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.932657 1166 ts_tablet_manager.cc:1905] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:25.933215 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0aeb7077591f466eb77e3096f6b1d74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aeb7077591f466eb77e3096f6b1d74b already in progress: creating tablet
I20250625 07:52:25.937865 753 tablet_service.cc:1468] Processing CreateTablet for tablet f1acadde98f54b65830c57df55b06ec3 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:25.938026 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5cbace0f99a44800bab1de5e38eb5112 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cbace0f99a44800bab1de5e38eb5112 already in progress: creating tablet
W20250625 07:52:25.938300 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b620bb670cd94dce999cbe1dabc6125a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b620bb670cd94dce999cbe1dabc6125a already in progress: creating tablet
I20250625 07:52:25.938516 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 450727d8f33e4eb489969b7974cdeeb1 (table default.Table_5c23014bc5574cc9a64e940e87188050 [id=01cb78211cdc4a9a9b493e31f8c44a99]) successfully deleted
I20250625 07:52:25.921978 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "494cacf2dcbb40d3b2c9f05f64809870" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:25.939179 850 leader_election.cc:343] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:25.939325 850 leader_election.cc:304] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:25.939591 1793 raft_consensus.cc:2747] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:25.940372 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1acadde98f54b65830c57df55b06ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.940920 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b (DEFAULT_TABLE table=default.Table_72eb667adefe48c582a543cd03322971 [id=1d231960767e497ba040a869d87c47d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.941295 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a653e84823d467f9beb897f2bfb3b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.941874 1166 ts_tablet_manager.cc:1918] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.941956 1166 log.cc:1199] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/445bfed532004108a12c8fd78370c1cb
I20250625 07:52:25.942214 1166 ts_tablet_manager.cc:1939] T 445bfed532004108a12c8fd78370c1cb P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:25.942646 746 tablet_service.cc:1468] Processing CreateTablet for tablet 731a30dc035146d584d11cbecd12baf0 (DEFAULT_TABLE table=default.Table_b9c59f89eb80476cb447870b7c2b4251 [id=aa60d477307744dab9a6a0762e86553a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.943048 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731a30dc035146d584d11cbecd12baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.949175 1687 server_base.cc:1048] running on GCE node
I20250625 07:52:25.951009 1162 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:25.951574 1799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:25.955571 1158 tablet_bootstrap.cc:492] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:25.955813 1158 ts_tablet_manager.cc:1397] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250625 07:52:25.956630 1158 raft_consensus.cc:357] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.956936 1158 raft_consensus.cc:383] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.957051 1158 raft_consensus.cc:738] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:25.957353 1158 consensus_queue.cc:260] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.958514 1158 ts_tablet_manager.cc:1428] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:25.959894 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b (DEFAULT_TABLE table=default.Table_72eb667adefe48c582a543cd03322971 [id=1d231960767e497ba040a869d87c47d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.960147 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1742634837e944ffb9cab3646b933950 (DEFAULT_TABLE table=default.Table_997a282eabfa48049916f38d2f6b7089 [id=d400a52f8fde4cddbf984d4eed3823de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.960350 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a653e84823d467f9beb897f2bfb3b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.961115 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 33441054063e47deaba2ad04ae6ae764 (DEFAULT_TABLE table=default.Table_f07796775e9f4e0299fa92332951f3f5 [id=b442bda0b9684e9c8daca6f8c82e84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.961618 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 2398a407b3154334a6a167946cf99ad7 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.961957 885 tablet_service.cc:1468] Processing CreateTablet for tablet 2398a407b3154334a6a167946cf99ad7 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.962177 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ccd929ffd026433c9abd3851bb8e1004 (DEFAULT_TABLE table=default.Table_f2029eb1ac264cc9967ec89a919d5431 [id=ca80dfc499bd48d3a56fcc8c8ce4cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.962316 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398a407b3154334a6a167946cf99ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.961400 1162 tablet_bootstrap.cc:654] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:25.962404 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5aa94b384aea4fedbba2d3431d7ee5f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.962993 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aa94b384aea4fedbba2d3431d7ee5f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.964460 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 445bfed532004108a12c8fd78370c1cb (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
I20250625 07:52:25.964927 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 9f30a8b59beb4311aca534ed322aff03 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.965078 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 768c9551d33346718af2670a01a626b0 (DEFAULT_TABLE table=default.Table_d86f89948c7043899036f6af6183d950 [id=f3a2c56d0fda43a5893c1992e177c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.963986 893 tablet_service.cc:1468] Processing CreateTablet for tablet ccd929ffd026433c9abd3851bb8e1004 (DEFAULT_TABLE table=default.Table_f2029eb1ac264cc9967ec89a919d5431 [id=ca80dfc499bd48d3a56fcc8c8ce4cf8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.965962 1013 tablet_service.cc:1468] Processing CreateTablet for tablet b7c5c675d0a7470e95a30a499ad15714 (DEFAULT_TABLE table=default.Table_014505d5378f4c5a89fefb3ada354601 [id=413c81222dbb445d9e5275e1e0797d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.965962 889 tablet_service.cc:1468] Processing CreateTablet for tablet b7c5c675d0a7470e95a30a499ad15714 (DEFAULT_TABLE table=default.Table_014505d5378f4c5a89fefb3ada354601 [id=413c81222dbb445d9e5275e1e0797d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.966097 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd929ffd026433c9abd3851bb8e1004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.966280 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c5c675d0a7470e95a30a499ad15714. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:25.966385 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d42f5f3910742f0942742eb77ea1435 consensus state has no leader
I20250625 07:52:25.966416 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4a30941e3b9c475db17f961c38d96ef4
I20250625 07:52:25.964881 895 tablet_service.cc:1468] Processing CreateTablet for tablet 768c9551d33346718af2670a01a626b0 (DEFAULT_TABLE table=default.Table_d86f89948c7043899036f6af6183d950 [id=f3a2c56d0fda43a5893c1992e177c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.966722 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 768c9551d33346718af2670a01a626b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.967801 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a6c0834adf1748c6a67805d26b973171
I20250625 07:52:25.968077 1160 ts_tablet_manager.cc:1918] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.968194 1160 log.cc:1199] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6239713a586d43c28c101ba2447d579d
I20250625 07:52:25.968550 1160 ts_tablet_manager.cc:1939] T 6239713a586d43c28c101ba2447d579d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:25.965701 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9f30a8b59beb4311aca534ed322aff03 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.969045 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f30a8b59beb4311aca534ed322aff03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.969867 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a6c0834adf1748c6a67805d26b973171 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.969846 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0139149bbf574e13be3b0411a83a0593 (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.970507 880 tablet_service.cc:1468] Processing CreateTablet for tablet 0139149bbf574e13be3b0411a83a0593 (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.970805 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0139149bbf574e13be3b0411a83a0593. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:25.972820 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a6c0834adf1748c6a67805d26b973171 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c0834adf1748c6a67805d26b973171 already in progress: creating tablet
I20250625 07:52:25.972954 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1742634837e944ffb9cab3646b933950 (DEFAULT_TABLE table=default.Table_997a282eabfa48049916f38d2f6b7089 [id=d400a52f8fde4cddbf984d4eed3823de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.973361 883 tablet_service.cc:1468] Processing CreateTablet for tablet 33441054063e47deaba2ad04ae6ae764 (DEFAULT_TABLE table=default.Table_f07796775e9f4e0299fa92332951f3f5 [id=b442bda0b9684e9c8daca6f8c82e84d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.973872 1733 raft_consensus.cc:491] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:25.974004 1733 raft_consensus.cc:513] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:25.974644 1733 leader_election.cc:290] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:25.974774 1162 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:25.974900 1162 ts_tablet_manager.cc:1397] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
W20250625 07:52:25.975353 985 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 76fe692067954856908f936ced332c18
I20250625 07:52:25.975757 1162 raft_consensus.cc:357] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.975914 1162 raft_consensus.cc:383] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:25.975958 1162 raft_consensus.cc:738] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:25.976133 986 leader_election.cc:343] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 76fe692067954856908f936ced332c18
I20250625 07:52:25.976238 986 leader_election.cc:304] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:25.976333 1162 consensus_queue.cc:260] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:25.976994 1733 raft_consensus.cc:2747] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:25.978138 1789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:25.979132 1162 ts_tablet_manager.cc:1428] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:25.980120 1162 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:25.980269 1160 tablet_replica.cc:331] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:25.982451 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c6fe509bff964fc783b30da1badc3431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:25.982802 1687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250625 07:52:25.982990 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6fe509bff964fc783b30da1badc3431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.984103 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb with delete_type TABLET_DATA_DELETED (replaced by tablet 29a2f1c24e0240c0b1f0007961ee3ad2) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.985170 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 47b4112b7e674518bcc1ea748370fd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.986439 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet cac7d20327e14f90af25acb58d724b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.987159 1012 tablet_service.cc:1468] Processing CreateTablet for tablet f1acadde98f54b65830c57df55b06ec3 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.988338 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:25.988549 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f5ab0400081848dd872a429685e3b8e2 (DEFAULT_TABLE table=default.Table_cebbbf28857c46a6adc44443c9a09488 [id=de815b21538b4d40a63a02ab538662bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.989228 1166 tablet_replica.cc:331] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:25.989477 1166 raft_consensus.cc:2241] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.989655 1166 raft_consensus.cc:2270] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.990155 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 7965eb5269ad44e0835c74d19fe23861 (DEFAULT_TABLE table=default.Table_3cede47fcd084b178dd44942300d7a4b [id=d1c10ec1fb964fc7bbd6fcf70aef6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.990631 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:25.989005 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8fcfcfd3d18a44818c59195ab07e9f57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:25.994261 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fcfcfd3d18a44818c59195ab07e9f57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.994577 1160 raft_consensus.cc:2241] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:25.994801 1160 raft_consensus.cc:2270] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:25.981024 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33441054063e47deaba2ad04ae6ae764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:25.997869 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 (DEFAULT_TABLE table=default.Table_c3a4e707895c4f80995d1f26e93afe45 [id=bffcae3d59bb4177ae9048be96e9fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.998867 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7c0a9001bc454c42b4f7a4713a27dee9" } new_table_name: "default.Table_1726e9aaa49544cf98ad7271dd414da8" modify_external_catalogs: true
I20250625 07:52:25.999282 1166 ts_tablet_manager.cc:1905] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.999282 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 731a30dc035146d584d11cbecd12baf0 (DEFAULT_TABLE table=default.Table_b9c59f89eb80476cb447870b7c2b4251 [id=aa60d477307744dab9a6a0762e86553a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.000391 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet d92962b9e7d14c8d88d775b98e75de35 with delete_type TABLET_DATA_DELETED (replaced by tablet 02ec8c9e6340497abe802c7345f42921) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.001268 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.002310 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a6c0834adf1748c6a67805d26b973171 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.003006 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6239713a586d43c28c101ba2447d579d (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
W20250625 07:52:26.003345 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b1cdde72919a4a3ca29434d475c2d44f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1cdde72919a4a3ca29434d475c2d44f already in progress: creating tablet
I20250625 07:52:26.006664 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_15eb4e6d76ae4533aeee6d180c721b13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.007264 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15eb4e6d76ae4533aeee6d180c721b13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:25.984760 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 77a127ebc7a74274b60cdd7d08ce22cb with delete_type TABLET_DATA_DELETED (replaced by tablet 29a2f1c24e0240c0b1f0007961ee3ad2) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.985631 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 47b4112b7e674518bcc1ea748370fd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.986037 884 tablet_service.cc:1515] Processing DeleteTablet for tablet cac7d20327e14f90af25acb58d724b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:25.987970 891 tablet_service.cc:1468] Processing CreateTablet for tablet f1acadde98f54b65830c57df55b06ec3 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.989259 888 tablet_service.cc:1468] Processing CreateTablet for tablet f5ab0400081848dd872a429685e3b8e2 (DEFAULT_TABLE table=default.Table_cebbbf28857c46a6adc44443c9a09488 [id=de815b21538b4d40a63a02ab538662bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.991261 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.010784 1687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:26.011493 1166 ts_tablet_manager.cc:1918] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:25.995498 1160 ts_tablet_manager.cc:1905] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:25.997280 879 tablet_service.cc:1468] Processing CreateTablet for tablet 7965eb5269ad44e0835c74d19fe23861 (DEFAULT_TABLE table=default.Table_3cede47fcd084b178dd44942300d7a4b [id=d1c10ec1fb964fc7bbd6fcf70aef6913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:25.997414 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1742634837e944ffb9cab3646b933950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.012037 1687 hybrid_clock.cc:648] HybridClock initialized: now 1750837946011993 us; error 39 us; skew 500 ppm
I20250625 07:52:25.998737 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 (DEFAULT_TABLE table=default.Table_c3a4e707895c4f80995d1f26e93afe45 [id=bffcae3d59bb4177ae9048be96e9fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.012460 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a91007503ad43cc8f6fae30e4206df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.012550 1687 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:25.999882 876 tablet_service.cc:1468] Processing CreateTablet for tablet 731a30dc035146d584d11cbecd12baf0 (DEFAULT_TABLE table=default.Table_b9c59f89eb80476cb447870b7c2b4251 [id=aa60d477307744dab9a6a0762e86553a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.013103 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731a30dc035146d584d11cbecd12baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.014178 1158 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.015832 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7965eb5269ad44e0835c74d19fe23861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.002103 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.016206 1158 tablet_bootstrap.cc:654] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.002862 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a6c0834adf1748c6a67805d26b973171 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.020227 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1acadde98f54b65830c57df55b06ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.020395 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33441054063e47deaba2ad04ae6ae764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.020903 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2398a407b3154334a6a167946cf99ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.009310 1163 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.007690 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.022643 1163 ts_tablet_manager.cc:1397] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.192s user 0.007s sys 0.026s
I20250625 07:52:26.024884 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd929ffd026433c9abd3851bb8e1004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.029134 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f30a8b59beb4311aca534ed322aff03. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.031018 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5de0c5bf60b24808af124c95833d9c25 consensus state has no leader
I20250625 07:52:26.016386 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a30941e3b9c475db17f961c38d96ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.031912 1166 log.cc:1199] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a0518fbc7f52454296ae79a3c92ef800
I20250625 07:52:26.032241 1166 ts_tablet_manager.cc:1939] T a0518fbc7f52454296ae79a3c92ef800 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:26.033272 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a0518fbc7f52454296ae79a3c92ef800 (table default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]) successfully deleted
I20250625 07:52:26.033753 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 768c9551d33346718af2670a01a626b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.033882 1158 tablet_bootstrap.cc:492] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.034035 1158 ts_tablet_manager.cc:1397] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20250625 07:52:26.034982 1158 raft_consensus.cc:357] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.035203 1158 raft_consensus.cc:383] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.035266 1158 raft_consensus.cc:738] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.035493 1158 consensus_queue.cc:260] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.036218 1158 ts_tablet_manager.cc:1428] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:26.031797 1162 tablet_bootstrap.cc:654] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.036548 1158 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.036994 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c5c675d0a7470e95a30a499ad15714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.037248 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1acadde98f54b65830c57df55b06ec3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.037813 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4a30941e3b9c475db17f961c38d96ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a30941e3b9c475db17f961c38d96ef4 already in progress: creating tablet
I20250625 07:52:26.038278 596 master_service.cc:948] ReplaceTablet: received request to replace tablet d1fc7cb613b6472ea34a0ea7bc8a08ea from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.040971 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ab0400081848dd872a429685e3b8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.044462 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4da3fcaa3520415aac448f633123fa14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.044926 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7965eb5269ad44e0835c74d19fe23861. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.045110 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4da3fcaa3520415aac448f633123fa14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.048530 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a91007503ad43cc8f6fae30e4206df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.052116 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 76fe692067954856908f936ced332c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.052945 1161 tablet_replica.cc:331] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:26.053117 1161 raft_consensus.cc:2241] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.053272 1161 raft_consensus.cc:2270] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.052132 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731a30dc035146d584d11cbecd12baf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.053761 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0139149bbf574e13be3b0411a83a0593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.057202 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1742634837e944ffb9cab3646b933950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.058267 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9a56722f526b4aceb8716901c46a8be5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.059020 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a56722f526b4aceb8716901c46a8be5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.061515 1161 ts_tablet_manager.cc:1905] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:26.065503 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 77a127ebc7a74274b60cdd7d08ce22cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77a127ebc7a74274b60cdd7d08ce22cb already in progress: creating tablet
W20250625 07:52:26.065801 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 47b4112b7e674518bcc1ea748370fd85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47b4112b7e674518bcc1ea748370fd85 already in progress: creating tablet
W20250625 07:52:26.066012 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet cac7d20327e14f90af25acb58d724b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7d20327e14f90af25acb58d724b68 already in progress: creating tablet
W20250625 07:52:26.066255 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0aeb7077591f466eb77e3096f6b1d74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aeb7077591f466eb77e3096f6b1d74b already in progress: creating tablet
I20250625 07:52:26.080536 1158 tablet_bootstrap.cc:654] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.096261 1687 webserver.cc:469] Webserver started at http://127.0.104.62:39255/ using document root <none> and password file <none>
I20250625 07:52:26.096333 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_929703cfa29e467c9090ff84a8b2cec1" } modify_external_catalogs: true
I20250625 07:52:26.096535 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_929703cfa29e467c9090ff84a8b2cec1" } modify_external_catalogs: true
I20250625 07:52:26.097656 1687 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:26.097864 1687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:26.098829 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3617e2588af44bbea69c7b4181f219e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.099436 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3617e2588af44bbea69c7b4181f219e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.114775 1687 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.005s sys 0.004s
I20250625 07:52:26.115240 1793 raft_consensus.cc:491] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.115933 1158 tablet_bootstrap.cc:492] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.116082 1158 ts_tablet_manager.cc:1397] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20250625 07:52:26.116870 1158 raft_consensus.cc:357] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.117035 1158 raft_consensus.cc:383] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.117090 1158 raft_consensus.cc:738] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.117326 1158 consensus_queue.cc:260] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.117892 1158 ts_tablet_manager.cc:1428] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:26.118750 1158 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.121168 1158 tablet_bootstrap.cc:654] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.130568 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dfe7357f10e546fa8c2a6bcc522beb1e" } new_table_name: "default.Table_444c92729289455eaf1a979360c97e5e" modify_external_catalogs: true
I20250625 07:52:26.132103 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1ef2001aff1f47f28501fb568a189e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.132745 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ef2001aff1f47f28501fb568a189e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.134016 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d907bad8a74a495a9bb4156ce049cd5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.134523 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d907bad8a74a495a9bb4156ce049cd5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.135653 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9f083e3394b84fcdae11d916769e9f57" } new_table_name: "default.Table_144dd070706a4d6e8d042f25ec83125e" modify_external_catalogs: true
I20250625 07:52:26.115443 1793 raft_consensus.cc:513] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.139146 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1cca330896134062a3528e0f5dca94c4" } new_table_name: "default.Table_7057fefe95d146c3a4dcf7de25f2d500" modify_external_catalogs: true
I20250625 07:52:26.139183 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c7cdd85509b14915a9668f890f2ed721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.139647 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7cdd85509b14915a9668f890f2ed721 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.142854 1809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:26.144955 1687 fs_manager.cc:730] Time spent opening block manager: real 0.028s user 0.002s sys 0.007s
I20250625 07:52:26.145223 1687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
uuid: "00a3faeffaa4422ea73b98ee74768a4a"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:26.146421 1687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:26.149418 1793 leader_election.cc:290] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.152204 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:26.152715 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:26.153198 850 leader_election.cc:343] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:26.153348 851 leader_election.cc:343] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.153452 851 leader_election.cc:304] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:26.153671 1793 raft_consensus.cc:2747] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:26.154871 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4da3fcaa3520415aac448f633123fa14" } modify_external_catalogs: true
I20250625 07:52:26.155077 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4da3fcaa3520415aac448f633123fa14" } modify_external_catalogs: true
W20250625 07:52:26.157063 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da895c90d6ee40379fe8181c2dbd7a1e consensus state has no leader
I20250625 07:52:26.157569 1163 raft_consensus.cc:357] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.157735 1163 raft_consensus.cc:383] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.157799 1163 raft_consensus.cc:738] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.158013 1163 consensus_queue.cc:260] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.158524 1163 ts_tablet_manager.cc:1428] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.136s user 0.002s sys 0.000s
I20250625 07:52:26.158833 1163 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.161221 1163 tablet_bootstrap.cc:654] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.171187 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fe4cc985dba64833998e95782e5634a9" } new_table_name: "default.Table_7d2d9da2b28f4e77a4369967441acd13" modify_external_catalogs: true
W20250625 07:52:26.173710 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2082e5e28c5c4d1fac7e40acf77f75e6 consensus state has no leader
I20250625 07:52:26.174232 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_53313429425a455cb97296e7d4b6f0dd" } modify_external_catalogs: true
I20250625 07:52:26.174437 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_53313429425a455cb97296e7d4b6f0dd" } modify_external_catalogs: true
I20250625 07:52:26.174757 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d77e4cd0cb1143258e09219d7910fc82" } modify_external_catalogs: true
I20250625 07:52:26.174932 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d77e4cd0cb1143258e09219d7910fc82" } modify_external_catalogs: true
I20250625 07:52:26.175776 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a4a61d6fd2514d88989f611b35c12baa" } modify_external_catalogs: true
I20250625 07:52:26.175904 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a4a61d6fd2514d88989f611b35c12baa" } modify_external_catalogs: true
W20250625 07:52:26.176502 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37fe97b46b874771ae5ca8c849720118 has no consensus state
I20250625 07:52:26.180181 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d14f9617cfa34c00ae651e78dc5be263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.180800 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d14f9617cfa34c00ae651e78dc5be263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.183318 592 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ca6f9c7909c045e5b36d9ec2e562ff19
I20250625 07:52:26.185326 1158 tablet_bootstrap.cc:492] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.185386 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ab0400081848dd872a429685e3b8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.185549 1158 ts_tablet_manager.cc:1397] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.002s
I20250625 07:52:26.186304 1158 raft_consensus.cc:357] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.186491 1158 raft_consensus.cc:383] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.186545 1158 raft_consensus.cc:738] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.186772 1158 consensus_queue.cc:260] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.187350 1158 ts_tablet_manager.cc:1428] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250625 07:52:26.187525 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 consensus state has no leader
I20250625 07:52:26.187629 1158 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.189059 1158 tablet_bootstrap.cc:654] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:26.192338 851 consensus_peers.cc:487] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 -> Peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Couldn't send request to peer 191b611b772a4fda9d1cba98776138a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e41e53efb6f342d2b1e4d642f7aaad10. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:26.193806 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.194414 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.200083 1161 ts_tablet_manager.cc:1918] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.200223 1161 log.cc:1199] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/d92962b9e7d14c8d88d775b98e75de35
I20250625 07:52:26.200564 1161 ts_tablet_manager.cc:1939] T d92962b9e7d14c8d88d775b98e75de35 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:26.201359 1161 tablet_replica.cc:331] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:26.201540 1161 raft_consensus.cc:2241] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.201699 1161 raft_consensus.cc:2270] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.201944 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce with delete_type TABLET_DATA_DELETED (replaced by tablet b8b5bb3fe3ef4bfdba93b862784a9f61) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.204363 1163 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.204625 1163 ts_tablet_manager.cc:1397] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20250625 07:52:26.205742 1163 raft_consensus.cc:357] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.206045 1163 raft_consensus.cc:383] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.206241 1163 raft_consensus.cc:738] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.206643 1163 consensus_queue.cc:260] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.207777 1163 ts_tablet_manager.cc:1428] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:26.208001 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet d92962b9e7d14c8d88d775b98e75de35 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]) successfully deleted
W20250625 07:52:26.208600 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4a30941e3b9c475db17f961c38d96ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a30941e3b9c475db17f961c38d96ef4 already in progress: creating tablet
W20250625 07:52:26.208936 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a6c0834adf1748c6a67805d26b973171 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c0834adf1748c6a67805d26b973171 already in progress: creating tablet
I20250625 07:52:26.209403 1163 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.212101 1163 tablet_bootstrap.cc:654] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.213910 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet ebb943df93284913b6a520e099399322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.223434 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d450a894040241c7bc3d4d34bca6a755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.224278 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d450a894040241c7bc3d4d34bca6a755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.226871 1161 ts_tablet_manager.cc:1905] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.229596 590 master_service.cc:948] ReplaceTablet: received request to replace tablet e73689f413504e28ba97a83ab7b03fa1 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.230589 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e40dd0454868478dadc2bc891d78f59c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.231225 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e40dd0454868478dadc2bc891d78f59c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.233916 1163 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.234058 1163 ts_tablet_manager.cc:1397] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20250625 07:52:26.234822 1163 raft_consensus.cc:357] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.234980 1163 raft_consensus.cc:383] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.235033 1163 raft_consensus.cc:738] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.235231 1163 consensus_queue.cc:260] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.236290 1163 ts_tablet_manager.cc:1428] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:26.238601 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a87d4f6703a94cbb96962177f2bb0cd7" } new_table_name: "default.Table_81f70c78489e42d2815ce9a638e5b156" modify_external_catalogs: true
I20250625 07:52:26.240571 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d357cefb84a239d5be73857b9097b (DEFAULT_TABLE table=default.Table_5614a7608ac14d9dbf64878ba87a66de [id=b4bb01b236b34c5d9e23dea3b8cf78d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.240980 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:26.242292 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d357cefb84a239d5be73857b9097b (DEFAULT_TABLE table=default.Table_5614a7608ac14d9dbf64878ba87a66de [id=b4bb01b236b34c5d9e23dea3b8cf78d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.242705 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d357cefb84a239d5be73857b9097b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.236677 1163 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.249722 883 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d357cefb84a239d5be73857b9097b (DEFAULT_TABLE table=default.Table_5614a7608ac14d9dbf64878ba87a66de [id=b4bb01b236b34c5d9e23dea3b8cf78d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.250142 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d357cefb84a239d5be73857b9097b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.254993 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:26.255709 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 02ec8c9e6340497abe802c7345f42921 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]). Expected version 5 got 0
I20250625 07:52:26.255393 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3c26e45d95314578aa155e978ffd1096"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.257081 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:26.257241 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c26e45d95314578aa155e978ffd1096 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.259812 1162 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.260020 1162 ts_tablet_manager.cc:1397] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.280s user 0.031s sys 0.046s
I20250625 07:52:26.260073 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1764c3d48a7546f7828ee93aa51dcb51
I20250625 07:52:26.261029 1162 raft_consensus.cc:357] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.261332 1162 raft_consensus.cc:383] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.261454 1162 raft_consensus.cc:738] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.261736 1162 consensus_queue.cc:260] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.261829 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.262389 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.264022 1162 ts_tablet_manager.cc:1428] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:26.264339 1162 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:26.264544 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1764c3d48a7546f7828ee93aa51dcb51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1764c3d48a7546f7828ee93aa51dcb51 already in progress: creating tablet
I20250625 07:52:26.264793 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 1764c3d48a7546f7828ee93aa51dcb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.267859 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.268831 1166 tablet_replica.cc:331] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:26.268998 1166 raft_consensus.cc:2241] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.269126 1166 raft_consensus.cc:2270] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.270043 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d357cefb84a239d5be73857b9097b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.272583 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b6661b294489423ab06fa7c6988b75f7
I20250625 07:52:26.272943 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 670f3b7f0a904656af418d01ed6f58b4
I20250625 07:52:26.273444 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d1fc7cb613b6472ea34a0ea7bc8a08ea
I20250625 07:52:26.273738 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.273862 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a630f8804bbf41de86b08cde419643ff
I20250625 07:52:26.274307 888 tablet_service.cc:1515] Processing DeleteTablet for tablet b6661b294489423ab06fa7c6988b75f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:26.274627 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e73689f413504e28ba97a83ab7b03fa1 consensus state has no leader
I20250625 07:52:26.273861 596 catalog_manager.cc:6652] ReplaceTablet: tablet d1fc7cb613b6472ea34a0ea7bc8a08ea deleted and replaced by tablet 8f7e4e88633c42a7b1bf70a4c64cea67
I20250625 07:52:26.275044 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet a630f8804bbf41de86b08cde419643ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.275260 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea with delete_type TABLET_DATA_DELETED (replaced by tablet 8f7e4e88633c42a7b1bf70a4c64cea67) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.275748 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea with delete_type TABLET_DATA_DELETED (replaced by tablet 8f7e4e88633c42a7b1bf70a4c64cea67) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.276122 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a630f8804bbf41de86b08cde419643ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.276345 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.276410 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a630f8804bbf41de86b08cde419643ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.276468 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.276911 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea with delete_type TABLET_DATA_DELETED (replaced by tablet 8f7e4e88633c42a7b1bf70a4c64cea67) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.276777 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b6661b294489423ab06fa7c6988b75f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.277673 1166 ts_tablet_manager.cc:1905] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.278613 1158 tablet_bootstrap.cc:492] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.278736 1158 ts_tablet_manager.cc:1397] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20250625 07:52:26.280336 1158 raft_consensus.cc:357] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.280485 1158 raft_consensus.cc:383] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.280536 1158 raft_consensus.cc:738] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.280843 1158 consensus_queue.cc:260] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.281350 1158 ts_tablet_manager.cc:1428] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:26.284499 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2ddae2af7aaf4116a475872071ad289e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.285166 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ddae2af7aaf4116a475872071ad289e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.286859 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e73689f413504e28ba97a83ab7b03fa1
I20250625 07:52:26.286913 753 tablet_service.cc:1515] Processing DeleteTablet for tablet b6661b294489423ab06fa7c6988b75f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.287338 590 catalog_manager.cc:6652] ReplaceTablet: tablet e73689f413504e28ba97a83ab7b03fa1 deleted and replaced by tablet 9a1e2434d6f04ac099d67bdc3ebdf335
I20250625 07:52:26.287592 753 tablet_service.cc:1515] Processing DeleteTablet for tablet e73689f413504e28ba97a83ab7b03fa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1e2434d6f04ac099d67bdc3ebdf335) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.293927 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e73689f413504e28ba97a83ab7b03fa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1e2434d6f04ac099d67bdc3ebdf335) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.294044 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa30ca0035048d3b60669994d402734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.294606 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e73689f413504e28ba97a83ab7b03fa1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1e2434d6f04ac099d67bdc3ebdf335) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.296841 1158 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.298836 1793 raft_consensus.cc:491] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.299053 1166 ts_tablet_manager.cc:1918] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.299000 1793 raft_consensus.cc:513] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.299149 1166 log.cc:1199] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:26.299443 1166 ts_tablet_manager.cc:1939] T 250175f54ba94a9fbc2455d9cf27fb2e P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:26.299659 1793 leader_election.cc:290] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.301908 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541c983d64f348539979f74199f79e59" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:26.302415 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541c983d64f348539979f74199f79e59" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:26.303299 851 leader_election.cc:343] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:26.303763 850 leader_election.cc:343] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.303869 850 leader_election.cc:304] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:26.304071 1793 raft_consensus.cc:2747] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:26.305953 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 250175f54ba94a9fbc2455d9cf27fb2e (table default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]) successfully deleted
I20250625 07:52:26.308148 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d715361336db4dcab94ab01d4f00ce32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.311039 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d715361336db4dcab94ab01d4f00ce32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.313419 888 tablet_service.cc:1468] Processing CreateTablet for tablet d4c6750037894d619a450a5dd9c1a1c7 (DEFAULT_TABLE table=default.Table_f6a3ec7a88e04f038252d1ceb4cdf7a7 [id=c1b79c9f46e84f7181b2981fb79bf2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.313848 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c6750037894d619a450a5dd9c1a1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.315963 753 tablet_service.cc:1468] Processing CreateTablet for tablet d4c6750037894d619a450a5dd9c1a1c7 (DEFAULT_TABLE table=default.Table_f6a3ec7a88e04f038252d1ceb4cdf7a7 [id=c1b79c9f46e84f7181b2981fb79bf2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.316390 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c6750037894d619a450a5dd9c1a1c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.316992 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a630f8804bbf41de86b08cde419643ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a630f8804bbf41de86b08cde419643ff already in progress: creating tablet
W20250625 07:52:26.317651 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 670f3b7f0a904656af418d01ed6f58b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670f3b7f0a904656af418d01ed6f58b4 already in progress: creating tablet
I20250625 07:52:26.321218 1163 tablet_bootstrap.cc:654] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:26.323690 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fc7cb613b6472ea34a0ea7bc8a08ea already in progress: creating tablet
W20250625 07:52:26.327076 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b6661b294489423ab06fa7c6988b75f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b294489423ab06fa7c6988b75f7 already in progress: creating tablet
W20250625 07:52:26.327337 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e73689f413504e28ba97a83ab7b03fa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e73689f413504e28ba97a83ab7b03fa1 already in progress: creating tablet
I20250625 07:52:26.328109 1162 tablet_bootstrap.cc:654] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.328181 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe97b46b874771ae5ca8c849720118 (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.328578 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe97b46b874771ae5ca8c849720118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.328905 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 2398a407b3154334a6a167946cf99ad7 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.298353 1158 tablet_bootstrap.cc:654] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.335793 1160 ts_tablet_manager.cc:1918] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:26.335918 1160 log.cc:1199] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8aa30ca0035048d3b60669994d402734
I20250625 07:52:26.335831 753 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe97b46b874771ae5ca8c849720118 (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.336261 1160 ts_tablet_manager.cc:1939] T 8aa30ca0035048d3b60669994d402734 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.336606 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe97b46b874771ae5ca8c849720118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.338991 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2398a407b3154334a6a167946cf99ad7
I20250625 07:52:26.339591 592 catalog_manager.cc:6652] ReplaceTablet: tablet 2398a407b3154334a6a167946cf99ad7 deleted and replaced by tablet 157590e34b8442b28e01f5c1c4b7d905
I20250625 07:52:26.339815 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 2398a407b3154334a6a167946cf99ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 157590e34b8442b28e01f5c1c4b7d905) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.340106 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2398a407b3154334a6a167946cf99ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 157590e34b8442b28e01f5c1c4b7d905) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.340385 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 2398a407b3154334a6a167946cf99ad7 with delete_type TABLET_DATA_DELETED (replaced by tablet 157590e34b8442b28e01f5c1c4b7d905) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:26.341252 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2398a407b3154334a6a167946cf99ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2398a407b3154334a6a167946cf99ad7 already in progress: creating tablet
I20250625 07:52:26.341307 592 master_service.cc:948] ReplaceTablet: received request to replace tablet a0342575a2b34bf4b2390968e6acbe5a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.341953 888 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe97b46b874771ae5ca8c849720118 (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.342415 1158 tablet_bootstrap.cc:492] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.342458 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe97b46b874771ae5ca8c849720118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.342564 1158 ts_tablet_manager.cc:1397] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20250625 07:52:26.349123 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a0342575a2b34bf4b2390968e6acbe5a
I20250625 07:52:26.349880 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a0342575a2b34bf4b2390968e6acbe5a with delete_type TABLET_DATA_DELETED (replaced by tablet 8d610d1c6d8f4f269387e56f099a782c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.350210 592 catalog_manager.cc:6652] ReplaceTablet: tablet a0342575a2b34bf4b2390968e6acbe5a deleted and replaced by tablet 8d610d1c6d8f4f269387e56f099a782c
I20250625 07:52:26.350597 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a0342575a2b34bf4b2390968e6acbe5a with delete_type TABLET_DATA_DELETED (replaced by tablet 8d610d1c6d8f4f269387e56f099a782c) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:26.351159 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a0342575a2b34bf4b2390968e6acbe5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0342575a2b34bf4b2390968e6acbe5a already in progress: creating tablet
I20250625 07:52:26.344574 1158 raft_consensus.cc:357] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.354420 1158 raft_consensus.cc:383] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.354480 1158 raft_consensus.cc:738] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.349797 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a0342575a2b34bf4b2390968e6acbe5a with delete_type TABLET_DATA_DELETED (replaced by tablet 8d610d1c6d8f4f269387e56f099a782c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.354640 746 tablet_service.cc:1468] Processing CreateTablet for tablet 660f49307d8f4435b7e1005a5d94de8d (DEFAULT_TABLE table=default.Table_b7f6776839094087962a6b3d39f30ad9 [id=817a8ec408b34df88deb1964fdc61f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.354889 1158 consensus_queue.cc:260] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.355108 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660f49307d8f4435b7e1005a5d94de8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.355508 1158 ts_tablet_manager.cc:1428] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 07:52:26.355820 1158 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.351380 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 660f49307d8f4435b7e1005a5d94de8d (DEFAULT_TABLE table=default.Table_b7f6776839094087962a6b3d39f30ad9 [id=817a8ec408b34df88deb1964fdc61f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.357321 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660f49307d8f4435b7e1005a5d94de8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.357410 1158 tablet_bootstrap.cc:654] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.358769 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_86dbdcd72c5a41f8adee048e24e3e9e8" } new_table_name: "default.Table_e5c7df9fdb6b4377988fdf47f909056c" modify_external_catalogs: true
I20250625 07:52:26.359288 1160 tablet_replica.cc:331] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.359483 1160 raft_consensus.cc:2241] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.359627 1160 raft_consensus.cc:2270] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.360211 1160 ts_tablet_manager.cc:1905] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.361516 1162 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.361652 1162 ts_tablet_manager.cc:1397] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.004s
I20250625 07:52:26.362426 1162 raft_consensus.cc:357] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.362586 1162 raft_consensus.cc:383] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.362648 1162 raft_consensus.cc:738] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.362866 1162 consensus_queue.cc:260] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:26.368494 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet babaf4dbc59147699c7b9a16c82e0774 consensus state has no leader
I20250625 07:52:26.370199 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:26.372419 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a5a3751333bc48fc8a59b0e7b67a9935 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5a3751333bc48fc8a59b0e7b67a9935 already in progress: creating tablet
I20250625 07:52:26.377341 1158 tablet_bootstrap.cc:492] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.377488 1158 ts_tablet_manager.cc:1397] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250625 07:52:26.378084 1158 raft_consensus.cc:357] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.378263 1158 raft_consensus.cc:383] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.378326 1158 raft_consensus.cc:738] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.378355 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8aa30ca0035048d3b60669994d402734 (table default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]) successfully deleted
I20250625 07:52:26.378589 1158 consensus_queue.cc:260] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.378938 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f62b03db94d4ee3935866b5dd8bd53f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.379070 1158 ts_tablet_manager.cc:1428] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:26.379326 1158 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.379364 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d3a57960634c41bd8b2bb81061f1a4ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.380035 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3a57960634c41bd8b2bb81061f1a4ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.384063 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_85849694e69e4737a9793e7c46e40a87" } new_table_name: "default.Table_4a4b98165527418ea3ccc846b86e4d15" modify_external_catalogs: true
I20250625 07:52:26.385499 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_48ce5f2a89b14013b3fcb44470c38912" } new_table_name: "default.Table_bf313c869fc24725bf68c339d6bd9c04" modify_external_catalogs: true
I20250625 07:52:26.385607 1014 tablet_service.cc:1468] Processing CreateTablet for tablet d4c6750037894d619a450a5dd9c1a1c7 (DEFAULT_TABLE table=default.Table_f6a3ec7a88e04f038252d1ceb4cdf7a7 [id=c1b79c9f46e84f7181b2981fb79bf2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.386029 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c6750037894d619a450a5dd9c1a1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.386819 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c0c5f7be7e344a1c9172a48d12556733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.387372 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c5f7be7e344a1c9172a48d12556733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.390013 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9bc3d032aee948de91f3c5c8c776cf7d" } modify_external_catalogs: true
I20250625 07:52:26.390177 1158 tablet_bootstrap.cc:654] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.390216 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9bc3d032aee948de91f3c5c8c776cf7d" } modify_external_catalogs: true
I20250625 07:52:26.390175 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2ad23fe585834c1aba8313557a4fa011" } new_table_name: "default.Table_032dd0d1cc4f45a685a90e4fb6c3d3a5" modify_external_catalogs: true
I20250625 07:52:26.392972 1160 ts_tablet_manager.cc:1918] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.393066 883 tablet_service.cc:1468] Processing CreateTablet for tablet 660f49307d8f4435b7e1005a5d94de8d (DEFAULT_TABLE table=default.Table_b7f6776839094087962a6b3d39f30ad9 [id=817a8ec408b34df88deb1964fdc61f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.393491 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 660f49307d8f4435b7e1005a5d94de8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.393594 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.394126 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.396744 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b4005dc5ede948c992182a74c6f1e2c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.397277 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4005dc5ede948c992182a74c6f1e2c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.397545 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.397941 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883c5592cc014b8eb7d85f62242e5b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.404716 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2f5b39a2075a487d8cbe4d4c9319f779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.405262 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5b39a2075a487d8cbe4d4c9319f779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:26.405499 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cdee1535e5a4a21804e28e78aaea0eb consensus state has no leader
W20250625 07:52:26.408281 713 consensus_peers.cc:487] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:26.393091 1160 log.cc:1199] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:26.409085 1160 ts_tablet_manager.cc:1939] T 494cacf2dcbb40d3b2c9f05f64809870 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.409945 1160 tablet_replica.cc:331] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.410118 1160 raft_consensus.cc:2241] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.410245 1160 raft_consensus.cc:2270] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.410696 1162 ts_tablet_manager.cc:1428] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250625 07:52:26.410751 1160 ts_tablet_manager.cc:1905] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.415531 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7e46f27344b848318dcd1e782819b75e
W20250625 07:52:26.416325 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e972cfec558b45e287ef91014173b74d consensus state has no leader
I20250625 07:52:26.418222 1793 raft_consensus.cc:491] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.418358 1793 raft_consensus.cc:513] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.418941 1793 leader_election.cc:290] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.419652 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 494cacf2dcbb40d3b2c9f05f64809870 (table default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]) successfully deleted
I20250625 07:52:26.419571 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcc3afc210a540058294121d2b83c66d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:26.420094 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e46f27344b848318dcd1e782819b75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.420117 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcc3afc210a540058294121d2b83c66d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:26.420485 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e6c2988baa14cbc8c878f79e942880d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:26.420740 851 leader_election.cc:343] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:26.420856 850 leader_election.cc:343] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.420981 850 leader_election.cc:304] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:26.421231 1793 raft_consensus.cc:2747] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:26.421269 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d42f5f3910742f0942742eb77ea1435 consensus state has no leader
I20250625 07:52:26.421711 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e46f27344b848318dcd1e782819b75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.424399 1687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:52:26.427888 595 master_service.cc:948] ReplaceTablet: received request to replace tablet f73561bac3bf4ece9da5433ab4d19b4e from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.429219 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e46f27344b848318dcd1e782819b75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.429916 893 tablet_service.cc:1468] Processing CreateTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:26.430145 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7e46f27344b848318dcd1e782819b75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e46f27344b848318dcd1e782819b75e already in progress: creating tablet
I20250625 07:52:26.430231 747 tablet_service.cc:1468] Processing CreateTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.430339 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883c5592cc014b8eb7d85f62242e5b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.430578 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 883c5592cc014b8eb7d85f62242e5b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.430773 1687 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:26.431476 1687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:26.436574 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f73561bac3bf4ece9da5433ab4d19b4e
I20250625 07:52:26.436990 595 catalog_manager.cc:6652] ReplaceTablet: tablet f73561bac3bf4ece9da5433ab4d19b4e deleted and replaced by tablet 308b387d366c4d7884898937a9fc7690
I20250625 07:52:26.437333 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.438071 747 tablet_service.cc:1515] Processing DeleteTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e with delete_type TABLET_DATA_DELETED (replaced by tablet 308b387d366c4d7884898937a9fc7690) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.438861 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e with delete_type TABLET_DATA_DELETED (replaced by tablet 308b387d366c4d7884898937a9fc7690) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.439296 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f73561bac3bf4ece9da5433ab4d19b4e with delete_type TABLET_DATA_DELETED (replaced by tablet 308b387d366c4d7884898937a9fc7690) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.439910 1161 ts_tablet_manager.cc:1918] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.440068 1161 log.cc:1199] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/76fe692067954856908f936ced332c18
I20250625 07:52:26.440456 1161 ts_tablet_manager.cc:1939] T 76fe692067954856908f936ced332c18 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:26.442348 1162 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:26.444032 1162 tablet_bootstrap.cc:654] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.444108 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 1221d470cf9344fa946db7bcc226992b (DEFAULT_TABLE table=default.Table_43f007c4611b415b8cdc3a9b8ab76966 [id=47a48ab6754641b4b62a8f57fa54281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.444512 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1221d470cf9344fa946db7bcc226992b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.444814 1166 tablet_replica.cc:331] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:26.444973 1166 raft_consensus.cc:2241] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.445133 1166 raft_consensus.cc:2270] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.447846 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1ac8b82938e14906a8a26fea4129029c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.448470 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ac8b82938e14906a8a26fea4129029c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.448534 1160 ts_tablet_manager.cc:1918] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.448633 1160 log.cc:1199] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7db110bb6cc146a393d62ddabbe1955a
I20250625 07:52:26.448966 1160 ts_tablet_manager.cc:1939] T 7db110bb6cc146a393d62ddabbe1955a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.450088 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7db110bb6cc146a393d62ddabbe1955a (table default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]) successfully deleted
I20250625 07:52:26.452095 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1221d470cf9344fa946db7bcc226992b (DEFAULT_TABLE table=default.Table_43f007c4611b415b8cdc3a9b8ab76966 [id=47a48ab6754641b4b62a8f57fa54281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.452304 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9e878ae7dd0c46c685ce00f99d31a339"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.452592 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1221d470cf9344fa946db7bcc226992b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.452979 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e878ae7dd0c46c685ce00f99d31a339 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.459105 1160 tablet_replica.cc:331] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.460278 755 tablet_service.cc:1468] Processing CreateTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.460505 1163 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.460633 1163 ts_tablet_manager.cc:1397] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.224s user 0.003s sys 0.000s
I20250625 07:52:26.460733 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a2f1c24e0240c0b1f0007961ee3ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.461328 1166 ts_tablet_manager.cc:1905] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.461673 1163 raft_consensus.cc:357] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.461886 1163 raft_consensus.cc:383] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.461947 1163 raft_consensus.cc:738] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.462318 1163 consensus_queue.cc:260] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.463526 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1ac8b82938e14906a8a26fea4129029c" } modify_external_catalogs: true
I20250625 07:52:26.463765 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1ac8b82938e14906a8a26fea4129029c" } modify_external_catalogs: true
I20250625 07:52:26.464993 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1221d470cf9344fa946db7bcc226992b (DEFAULT_TABLE table=default.Table_43f007c4611b415b8cdc3a9b8ab76966 [id=47a48ab6754641b4b62a8f57fa54281f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.465577 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1221d470cf9344fa946db7bcc226992b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.468113 1166 ts_tablet_manager.cc:1918] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.468179 1163 ts_tablet_manager.cc:1428] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:26.468225 1166 log.cc:1199] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:26.468508 1166 ts_tablet_manager.cc:1939] T a5a3751333bc48fc8a59b0e7b67a9935 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:26.468565 1163 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.474798 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a5a3751333bc48fc8a59b0e7b67a9935 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
I20250625 07:52:26.475703 1161 tablet_replica.cc:331] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:26.475941 1161 raft_consensus.cc:2241] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.476192 1161 raft_consensus.cc:2270] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.478085 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe117874dda4408934ce4e1687c10d6 (DEFAULT_TABLE table=default.Table_6f19a0a49fd04cffa91026e8f47a8bc7 [id=443b4cda475041c2a2f7969e937398a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.478502 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe117874dda4408934ce4e1687c10d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.482565 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 76fe692067954856908f936ced332c18 (table default.Table_33aedac05c3649c0a21c0367c4ff59a3 [id=b02af454e5f6454885578ace6c10ad97]) successfully deleted
I20250625 07:52:26.482731 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe117874dda4408934ce4e1687c10d6 (DEFAULT_TABLE table=default.Table_6f19a0a49fd04cffa91026e8f47a8bc7 [id=443b4cda475041c2a2f7969e937398a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.483088 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe117874dda4408934ce4e1687c10d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.484017 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e6092e518ceb4af7a5d84435008334ba" } modify_external_catalogs: true
I20250625 07:52:26.484123 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4006274930664fdca4ebf55f2375b00d" } new_table_name: "default.Table_4bb938fd70714708b7c8359750b8c076" modify_external_catalogs: true
I20250625 07:52:26.484221 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e6092e518ceb4af7a5d84435008334ba" } modify_external_catalogs: true
I20250625 07:52:26.485188 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_41047b79cb784bb19f458dbce4573ef5" } modify_external_catalogs: true
I20250625 07:52:26.485333 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_41047b79cb784bb19f458dbce4573ef5" } modify_external_catalogs: true
I20250625 07:52:26.485608 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6d71f14d5bd44891a4c2e98f3f43034e" } modify_external_catalogs: true
I20250625 07:52:26.485764 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6d71f14d5bd44891a4c2e98f3f43034e" } modify_external_catalogs: true
I20250625 07:52:26.491252 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.491664 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a2f1c24e0240c0b1f0007961ee3ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.491720 1163 tablet_bootstrap.cc:654] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.493510 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3f977dd0493949a8afda3f376504f89b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.493901 1160 raft_consensus.cc:2241] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.494130 1160 raft_consensus.cc:2270] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:26.494239 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f977dd0493949a8afda3f376504f89b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.494803 1160 ts_tablet_manager.cc:1905] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.497298 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 29c5b58dce124520a0f15e40b85b0dbc
W20250625 07:52:26.497681 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5fe3ea5745743019d23d49ff808070a consensus state has no leader
I20250625 07:52:26.498692 889 tablet_service.cc:1468] Processing CreateTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 (DEFAULT_TABLE table=default.Table_f426b18fff7f499db91894562dc4736e [id=f12791b214dd47e39d9dc3365f587e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.499073 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a2f1c24e0240c0b1f0007961ee3ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.508395 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b1909d0c3c834ca28b60d078f5c27129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.509018 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1909d0c3c834ca28b60d078f5c27129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.510499 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f5542f74374e4efe9acc6adc2c6e5b46
I20250625 07:52:26.510588 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet daf115dd3aa548dc802ed0362c274d53
I20250625 07:52:26.511500 742 tablet_service.cc:1515] Processing DeleteTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.511901 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 22411725aa5741c497328ce047c44d50
I20250625 07:52:26.511907 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.511956 749 tablet_service.cc:1515] Processing DeleteTablet for tablet daf115dd3aa548dc802ed0362c274d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.512116 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet daf115dd3aa548dc802ed0362c274d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.512903 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 22411725aa5741c497328ce047c44d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:26.513167 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f5542f74374e4efe9acc6adc2c6e5b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5542f74374e4efe9acc6adc2c6e5b46 already in progress: creating tablet
W20250625 07:52:26.513442 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet daf115dd3aa548dc802ed0362c274d53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf115dd3aa548dc802ed0362c274d53 already in progress: creating tablet
I20250625 07:52:26.513660 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 22411725aa5741c497328ce047c44d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.514464 895 tablet_service.cc:1515] Processing DeleteTablet for tablet daf115dd3aa548dc802ed0362c274d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:26.514931 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 22411725aa5741c497328ce047c44d50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22411725aa5741c497328ce047c44d50 already in progress: creating tablet
I20250625 07:52:26.515532 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 22411725aa5741c497328ce047c44d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.515780 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f5542f74374e4efe9acc6adc2c6e5b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.516461 1158 tablet_bootstrap.cc:492] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.516917 1158 ts_tablet_manager.cc:1397] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.009s
I20250625 07:52:26.517834 1158 raft_consensus.cc:357] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.518065 1158 raft_consensus.cc:383] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.518136 1158 raft_consensus.cc:738] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.518437 1158 consensus_queue.cc:260] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.519693 1161 ts_tablet_manager.cc:1905] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.520828 1158 ts_tablet_manager.cc:1428] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:26.521143 1158 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.522511 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c770b41e0844786ad81c7b3579f9934 with delete_type TABLET_DATA_DELETED (replaced by tablet cd728ffc2a274deea3fad533d1eaa6da) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.522835 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.529220 749 tablet_service.cc:1468] Processing CreateTablet for tablet c33b8be28b684f4babfa27e829bb64cf (DEFAULT_TABLE table=default.Table_defbe4d5e7684ec5ac3c36f8d600909c [id=10566617231a403bbf8d05fac0846fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.529458 1011 tablet_service.cc:1468] Processing CreateTablet for tablet c33b8be28b684f4babfa27e829bb64cf (DEFAULT_TABLE table=default.Table_defbe4d5e7684ec5ac3c36f8d600909c [id=10566617231a403bbf8d05fac0846fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.529758 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33b8be28b684f4babfa27e829bb64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.531235 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0a97d49e3a2f4a5bad206e655442709e (DEFAULT_TABLE table=default.Table_c776bde0fb9d4cdb9ed4ae90f85b078b [id=1d002346a75246a98db84ffb04ce7b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.534775 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b6ed2b0b8a3c43d2a87a016f9d8464f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.535382 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6ed2b0b8a3c43d2a87a016f9d8464f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.535883 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe117874dda4408934ce4e1687c10d6 (DEFAULT_TABLE table=default.Table_6f19a0a49fd04cffa91026e8f47a8bc7 [id=443b4cda475041c2a2f7969e937398a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.536549 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe117874dda4408934ce4e1687c10d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.538576 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a97d49e3a2f4a5bad206e655442709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.547708 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33b8be28b684f4babfa27e829bb64cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:26.554373 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 (table default.Table_7d2d9da2b28f4e77a4369967441acd13 [id=5e69b19f12c144e194afdc5ffc0cccce], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 consensus state has no leader
I20250625 07:52:26.558064 1158 tablet_bootstrap.cc:654] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.568445 1163 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.568619 1163 ts_tablet_manager.cc:1397] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20250625 07:52:26.569739 1163 raft_consensus.cc:357] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.569975 1163 raft_consensus.cc:383] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.570024 1163 raft_consensus.cc:738] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.570295 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4b28648f05f54681a7b82f8429d849ef" } new_table_name: "default.Table_2955e69ef88144b3963e9e522c6dd2a8" modify_external_catalogs: true
I20250625 07:52:26.570307 1163 consensus_queue.cc:260] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.576337 1163 ts_tablet_manager.cc:1428] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250625 07:52:26.576474 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96c58bffafa146ffacba18cf615c74b3 consensus state has no leader
I20250625 07:52:26.587049 888 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.592985 1163 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.595499 1163 tablet_bootstrap.cc:654] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.596423 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0a97d49e3a2f4a5bad206e655442709e (DEFAULT_TABLE table=default.Table_c776bde0fb9d4cdb9ed4ae90f85b078b [id=1d002346a75246a98db84ffb04ce7b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.596896 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a97d49e3a2f4a5bad206e655442709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.601142 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ddf70e7ccb3644f4b28ceb97decd744a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.601732 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddf70e7ccb3644f4b28ceb97decd744a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.601811 883 tablet_service.cc:1468] Processing CreateTablet for tablet c33b8be28b684f4babfa27e829bb64cf (DEFAULT_TABLE table=default.Table_defbe4d5e7684ec5ac3c36f8d600909c [id=10566617231a403bbf8d05fac0846fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.602164 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33b8be28b684f4babfa27e829bb64cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.607509 1158 tablet_bootstrap.cc:492] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.607657 1158 ts_tablet_manager.cc:1397] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20250625 07:52:26.608443 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8b2d59eebc0448e78ca54efc7537619e" } new_table_name: "default.Table_1c27389c187843de9945c7a9e3d1bed4" modify_external_catalogs: true
I20250625 07:52:26.608464 1158 raft_consensus.cc:357] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.608624 1158 raft_consensus.cc:383] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.608675 1158 raft_consensus.cc:738] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.608986 1158 consensus_queue.cc:260] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.610113 1158 ts_tablet_manager.cc:1428] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:26.616027 1163 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.616173 1163 ts_tablet_manager.cc:1397] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20250625 07:52:26.617537 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_125f22225a154f1288ab7309b9522d36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.618043 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_125f22225a154f1288ab7309b9522d36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.619174 1163 raft_consensus.cc:357] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.619345 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a1fac047060649b39adb2f5958d9816f" } modify_external_catalogs: true
I20250625 07:52:26.619405 1163 raft_consensus.cc:383] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.619467 1163 raft_consensus.cc:738] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.619527 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a1fac047060649b39adb2f5958d9816f" } modify_external_catalogs: true
I20250625 07:52:26.619676 1163 consensus_queue.cc:260] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.602640 1160 ts_tablet_manager.cc:1918] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.619999 1160 log.cc:1199] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2771159858fc41a1ad930229986edd92
I20250625 07:52:26.620391 1160 ts_tablet_manager.cc:1939] T 2771159858fc41a1ad930229986edd92 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.621800 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0ef24d5b4b124360b654d6903b994151" } new_table_name: "default.Table_9a7acbd7be614574a81e31a0f68bdbed" modify_external_catalogs: true
I20250625 07:52:26.622495 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f1b35f1d0a5a4b7db73348f6d04c46f7" } new_table_name: "default.Table_2acd2b198c4743d5a1ed1b69991d0450" modify_external_catalogs: true
I20250625 07:52:26.624598 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_146999741edf4bde91a875f03e40a74e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.624704 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_76ef6d59f0be4bc4804786565f912525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.625097 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_146999741edf4bde91a875f03e40a74e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:26.625203 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76ef6d59f0be4bc4804786565f912525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.632907 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8f09c9fd1b3a4dae9c8b366000737b5e
I20250625 07:52:26.634295 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.635181 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.635509 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f09c9fd1b3a4dae9c8b366000737b5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:26.637053 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8f09c9fd1b3a4dae9c8b366000737b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f09c9fd1b3a4dae9c8b366000737b5e already in progress: creating tablet
I20250625 07:52:26.638116 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0a97d49e3a2f4a5bad206e655442709e (DEFAULT_TABLE table=default.Table_c776bde0fb9d4cdb9ed4ae90f85b078b [id=1d002346a75246a98db84ffb04ce7b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:26.638579 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a97d49e3a2f4a5bad206e655442709e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:26.643668 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_51136d4037744ddfbd2b299213db9abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.643930 1158 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:26.644253 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51136d4037744ddfbd2b299213db9abd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.647145 1158 tablet_bootstrap.cc:654] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.659719 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f1f321543e7342108ad8b10b3c846822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.660302 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1f321543e7342108ad8b10b3c846822 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.661734 1163 ts_tablet_manager.cc:1428] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20250625 07:52:26.663240 1543 consensus_queue.cc:1035] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:26.664731 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet b6382963863548398bed90f03b840806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.665409 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 797) took 1052 ms. Trace:
I20250625 07:52:26.665539 1160 rpcz_store.cc:276] 0625 07:52:25.612808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.613807 (+ 999us) service_pool.cc:225] Handling call
0625 07:52:26.458772 (+844965us) ts_tablet_manager.cc:1145] Deleting tablet 2771159858fc41a1ad930229986edd92
0625 07:52:26.458791 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:26.598425 (+139634us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.602539 (+ 4114us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.665380 (+ 62841us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12241,"tablet-delete.queue_time_us":844727}
I20250625 07:52:26.666182 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2771159858fc41a1ad930229986edd92 (table default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]) successfully deleted
I20250625 07:52:26.667042 889 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.667243 1162 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.667404 1162 ts_tablet_manager.cc:1397] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.225s user 0.017s sys 0.005s
I20250625 07:52:26.668176 1162 raft_consensus.cc:357] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.668344 1162 raft_consensus.cc:383] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.668412 1162 raft_consensus.cc:738] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.668608 1162 consensus_queue.cc:260] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.669082 1162 ts_tablet_manager.cc:1428] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:26.669425 1162 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:26.669863 1161 ts_tablet_manager.cc:1918] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.673838 1162 tablet_bootstrap.cc:654] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.680299 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.680565 1166 tablet_replica.cc:331] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:26.680758 1166 raft_consensus.cc:2241] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.680902 1166 raft_consensus.cc:2270] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.686626 1163 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.689767 1166 ts_tablet_manager.cc:1905] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.691371 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eab5eda71c3145cda2327c5ffb28fb81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.691951 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eab5eda71c3145cda2327c5ffb28fb81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.692503 1160 tablet_replica.cc:331] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.694676 1160 raft_consensus.cc:2241] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.694862 1160 raft_consensus.cc:2270] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.695436 1160 ts_tablet_manager.cc:1905] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.699949 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_81f70c78489e42d2815ce9a638e5b156" } new_table_name: "default.Table_7b42526a3fb543b894d40b54df370f55" modify_external_catalogs: true
I20250625 07:52:26.701704 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_31f10020a22b49bc8778c19390b27fd0" } modify_external_catalogs: true
I20250625 07:52:26.701947 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_31f10020a22b49bc8778c19390b27fd0" } modify_external_catalogs: true
I20250625 07:52:26.704803 1161 log.cc:1199] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3c0edbf6cbdb49a6b3d3fff7f347c8ce
I20250625 07:52:26.705294 1161 ts_tablet_manager.cc:1939] T 3c0edbf6cbdb49a6b3d3fff7f347c8ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:26.705957 1161 tablet_replica.cc:331] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:26.706159 1161 raft_consensus.cc:2241] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.706319 1161 raft_consensus.cc:2270] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.706490 1158 tablet_bootstrap.cc:492] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.706622 1158 ts_tablet_manager.cc:1397] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.063s user 0.000s sys 0.004s
I20250625 07:52:26.706923 1161 ts_tablet_manager.cc:1905] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.707372 1158 raft_consensus.cc:357] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.707561 1158 raft_consensus.cc:383] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.707623 1158 raft_consensus.cc:738] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.707811 1158 consensus_queue.cc:260] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:26.708165 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a1e2434d6f04ac099d67bdc3ebdf335 has no consensus state
I20250625 07:52:26.715837 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 49697a10cdac4c8ebb69f10f4c3437ec with delete_type TABLET_DATA_DELETED (replaced by tablet a41ffd566593499299aba5513ccb2f30) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.718497 1166 ts_tablet_manager.cc:1918] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.718649 1166 log.cc:1199] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/29f2997d1d714100950383e6af04f315
I20250625 07:52:26.718981 1166 ts_tablet_manager.cc:1939] T 29f2997d1d714100950383e6af04f315 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:26.719272 1158 ts_tablet_manager.cc:1428] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250625 07:52:26.720785 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3c0edbf6cbdb49a6b3d3fff7f347c8ce (table default.Table_457927be3bfb45d1ab90fc5a3b8163d8 [id=9511cff29b2441689b47351734929304]) successfully deleted
I20250625 07:52:26.720894 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a3226ecfd38140ef85bca57822e323e1" } modify_external_catalogs: true
I20250625 07:52:26.721088 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a3226ecfd38140ef85bca57822e323e1" } modify_external_catalogs: true
I20250625 07:52:26.721518 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca8f9a686ffb42c18e83701403816a41" } modify_external_catalogs: true
I20250625 07:52:26.721655 1162 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.721773 1162 ts_tablet_manager.cc:1397] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.002s
I20250625 07:52:26.722224 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca8f9a686ffb42c18e83701403816a41" } modify_external_catalogs: true
I20250625 07:52:26.722522 1162 raft_consensus.cc:357] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.722667 1162 raft_consensus.cc:383] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.722713 1162 raft_consensus.cc:738] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.722905 1162 consensus_queue.cc:260] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.724310 1158 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.724658 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8c2ced3d2fba4f74a5d89d4bc5580905
I20250625 07:52:26.725931 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.729281 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b519d271b96497dafe0b274a8755066" } modify_external_catalogs: true
I20250625 07:52:26.729506 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b519d271b96497dafe0b274a8755066" } modify_external_catalogs: true
I20250625 07:52:26.732424 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 29f2997d1d714100950383e6af04f315 (table default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]) successfully deleted
I20250625 07:52:26.733008 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.735709 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.737111 1162 ts_tablet_manager.cc:1428] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20250625 07:52:26.738677 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_77d400c9d724449c8fb62b8761fb2784"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.739460 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77d400c9d724449c8fb62b8761fb2784 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.741871 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f5b05c08da4a4058ad3334ce9b0a8a84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.742563 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5b05c08da4a4058ad3334ce9b0a8a84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.743475 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.752265 1162 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:26.752377 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d605113ad8f246148825923e5bcdf00d
I20250625 07:52:26.753449 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet d605113ad8f246148825923e5bcdf00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.754402 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca22b663c92246f69b5beb7072264679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.755125 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca22b663c92246f69b5beb7072264679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.755571 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d605113ad8f246148825923e5bcdf00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.759238 588 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:26.759485 1162 tablet_bootstrap.cc:654] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:26.763545 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f73561bac3bf4ece9da5433ab4d19b4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73561bac3bf4ece9da5433ab4d19b4e already in progress: creating tablet
W20250625 07:52:26.765923 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c2ced3d2fba4f74a5d89d4bc5580905 already in progress: creating tablet
I20250625 07:52:26.767024 883 tablet_service.cc:1515] Processing DeleteTablet for tablet d605113ad8f246148825923e5bcdf00d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.769407 1166 tablet_replica.cc:331] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:26.770537 1166 raft_consensus.cc:2241] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.771660 1166 raft_consensus.cc:2270] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.774020 1163 tablet_bootstrap.cc:654] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.776978 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c8c015c202544914b74f1e245b496f82
I20250625 07:52:26.778164 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet c8c015c202544914b74f1e245b496f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:26.778414 755 tablet_service.cc:1515] Processing DeleteTablet for tablet c8c015c202544914b74f1e245b496f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.779723 1166 ts_tablet_manager.cc:1905] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.780319 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0aff505ed58e44ffaaea42615be646d1" } new_table_name: "default.Table_144eb7a6f3a1438294332f0eded4a470" modify_external_catalogs: true
I20250625 07:52:26.783329 883 tablet_service.cc:1515] Processing DeleteTablet for tablet c8c015c202544914b74f1e245b496f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.787686 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:26.793722 1163 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.793882 1163 ts_tablet_manager.cc:1397] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.002s
I20250625 07:52:26.794802 1163 raft_consensus.cc:357] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.794981 1163 raft_consensus.cc:383] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.795038 1163 raft_consensus.cc:738] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.795290 1163 consensus_queue.cc:260] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.799324 1162 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.799554 1162 ts_tablet_manager.cc:1397] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20250625 07:52:26.800366 1162 raft_consensus.cc:357] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.800597 1158 tablet_bootstrap.cc:654] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.800604 1162 raft_consensus.cc:383] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.800796 1162 raft_consensus.cc:738] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.801116 1162 consensus_queue.cc:260] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.801666 1162 ts_tablet_manager.cc:1428] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:26.802692 1163 ts_tablet_manager.cc:1428] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 07:52:26.806293 1163 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.810621 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4e7e572213414b8aa45a1abf1915a26d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.811213 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e7e572213414b8aa45a1abf1915a26d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:26.815403 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e109c9dca7444681ac517c7fc3e14dfa consensus state has no leader
I20250625 07:52:26.817303 1160 ts_tablet_manager.cc:1918] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.817986 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_616a034f739349138890dc31bd1d19c1" } new_table_name: "default.Table_8240eda72dae4b56a2f9173041146e83" modify_external_catalogs: true
I20250625 07:52:26.819190 1160 log.cc:1199] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:26.819581 1160 ts_tablet_manager.cc:1939] T 743123ffcf8240dc82bea7178559b7ea P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.820331 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 798) took 1205 ms. Trace:
I20250625 07:52:26.824127 1160 rpcz_store.cc:276] 0625 07:52:25.614630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.615343 (+ 713us) service_pool.cc:225] Handling call
0625 07:52:26.692436 (+1077093us) ts_tablet_manager.cc:1145] Deleting tablet 743123ffcf8240dc82bea7178559b7ea
0625 07:52:26.692451 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:26.789587 (+ 97136us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.817187 (+ 27600us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.820312 (+ 3125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076717}
I20250625 07:52:26.824726 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 743123ffcf8240dc82bea7178559b7ea (table default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]) successfully deleted
I20250625 07:52:26.819097 1163 tablet_bootstrap.cc:654] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.825989 1160 tablet_replica.cc:331] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.826268 1160 raft_consensus.cc:2241] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:26.826654 1160 raft_consensus.cc:2270] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.827227 1160 ts_tablet_manager.cc:1905] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:26.828186 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73ec635d7d5242e8804dd3527457a540 consensus state has no leader
I20250625 07:52:26.829103 1161 ts_tablet_manager.cc:1918] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.829285 1161 log.cc:1199] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/ebb943df93284913b6a520e099399322
I20250625 07:52:26.830312 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_397ddcb2840845c48bc37edf6de92bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.831089 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_397ddcb2840845c48bc37edf6de92bbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.835758 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.836160 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_72eb667adefe48c582a543cd03322971" } new_table_name: "default.Table_0885961127594fcb9cb80fba76deceee" modify_external_catalogs: true
I20250625 07:52:26.829676 1161 ts_tablet_manager.cc:1939] T ebb943df93284913b6a520e099399322 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:26.839154 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet ebb943df93284913b6a520e099399322 (table default.Table_014f0dfc938843338f50edc64d69b63a [id=4fd292fd56984d289b75b96de9d5613e]) successfully deleted
W20250625 07:52:26.839869 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1764c3d48a7546f7828ee93aa51dcb51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1764c3d48a7546f7828ee93aa51dcb51 already in progress: creating tablet
W20250625 07:52:26.840500 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 670f3b7f0a904656af418d01ed6f58b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670f3b7f0a904656af418d01ed6f58b4 already in progress: creating tablet
W20250625 07:52:26.841573 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a630f8804bbf41de86b08cde419643ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a630f8804bbf41de86b08cde419643ff already in progress: creating tablet
W20250625 07:52:26.842180 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fc7cb613b6472ea34a0ea7bc8a08ea already in progress: creating tablet
W20250625 07:52:26.842773 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b6661b294489423ab06fa7c6988b75f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b294489423ab06fa7c6988b75f7 already in progress: creating tablet
W20250625 07:52:26.843393 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e73689f413504e28ba97a83ab7b03fa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e73689f413504e28ba97a83ab7b03fa1 already in progress: creating tablet
I20250625 07:52:26.843701 1161 tablet_replica.cc:331] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:26.843711 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3cede47fcd084b178dd44942300d7a4b" } new_table_name: "default.Table_575db5f0ed644fecb44f5defd8fb2365" modify_external_catalogs: true
I20250625 07:52:26.843935 1161 raft_consensus.cc:2241] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.844125 1161 raft_consensus.cc:2270] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.844929 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8fcfcfd3d18a44818c59195ab07e9f57" } modify_external_catalogs: true
I20250625 07:52:26.845093 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8fcfcfd3d18a44818c59195ab07e9f57" } modify_external_catalogs: true
I20250625 07:52:26.846597 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7b26265e9f9a44bcbd5ba18263f61864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.847119 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b26265e9f9a44bcbd5ba18263f61864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.848616 1163 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.848785 1163 ts_tablet_manager.cc:1397] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.003s
I20250625 07:52:26.848763 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_329926f2f3e8457abd8548caa962293c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.849251 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_329926f2f3e8457abd8548caa962293c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.849704 1163 raft_consensus.cc:357] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.849892 1163 raft_consensus.cc:383] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.849948 1163 raft_consensus.cc:738] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.850173 1163 consensus_queue.cc:260] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.853706 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bdc29c3cb4b44fdabdaa413f46834dcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.854184 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdc29c3cb4b44fdabdaa413f46834dcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.855710 1163 ts_tablet_manager.cc:1428] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:26.858480 883 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.844660 1161 ts_tablet_manager.cc:1905] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.861863 1163 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.866097 1163 tablet_bootstrap.cc:654] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.869751 1158 tablet_bootstrap.cc:492] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:26.869959 1158 ts_tablet_manager.cc:1397] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.146s user 0.006s sys 0.007s
I20250625 07:52:26.870733 1158 raft_consensus.cc:357] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.870968 1158 raft_consensus.cc:383] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.871093 1158 raft_consensus.cc:738] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.871646 1158 consensus_queue.cc:260] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:26.872244 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a653e84823d467f9beb897f2bfb3b3b consensus state has no leader
I20250625 07:52:26.872570 1160 ts_tablet_manager.cc:1918] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:26.872676 1160 log.cc:1199] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e41e53efb6f342d2b1e4d642f7aaad10
I20250625 07:52:26.873018 1160 ts_tablet_manager.cc:1939] T e41e53efb6f342d2b1e4d642f7aaad10 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:26.875161 1158 ts_tablet_manager.cc:1428] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:26.875468 1793 raft_consensus.cc:491] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.875624 1793 raft_consensus.cc:513] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.875990 1158 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:26.876291 1793 leader_election.cc:290] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.877200 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9a94dc1432d4cbdbf3eecacb1429c23" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:26.877331 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9a94dc1432d4cbdbf3eecacb1429c23" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:26.877749 850 leader_election.cc:343] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:26.878170 851 leader_election.cc:343] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.878288 851 leader_election.cc:304] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:26.878708 1793 raft_consensus.cc:2747] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:26.879130 1158 tablet_bootstrap.cc:654] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.879730 1162 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:26.882349 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 794) took 1279 ms. Trace:
I20250625 07:52:26.882485 1160 rpcz_store.cc:276] 0625 07:52:25.603117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.616959 (+ 13842us) service_pool.cc:225] Handling call
0625 07:52:26.825942 (+1208983us) ts_tablet_manager.cc:1145] Deleting tablet e41e53efb6f342d2b1e4d642f7aaad10
0625 07:52:26.825952 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:26.832576 (+ 6624us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.872462 (+ 39886us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.882328 (+ 9866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207967}
W20250625 07:52:26.884593 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7965eb5269ad44e0835c74d19fe23861 consensus state has no leader
I20250625 07:52:26.884838 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]) successfully deleted
I20250625 07:52:26.885125 1160 tablet_replica.cc:331] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.885668 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ecd308318d99472b929b8e811d4d613f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.886667 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecd308318d99472b929b8e811d4d613f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.887046 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 2 got 0
I20250625 07:52:26.887907 1160 raft_consensus.cc:2241] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.888204 1160 raft_consensus.cc:2270] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.890507 595 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 2 got 0
I20250625 07:52:26.888749 1160 ts_tablet_manager.cc:1905] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.892596 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 6606a9559e6b47d28669083af5aee5bd from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:26.892652 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 110176us
I20250625 07:52:26.893385 1048 raft_consensus.cc:3058] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:26.898288 1163 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.898433 1163 ts_tablet_manager.cc:1397] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250625 07:52:26.899415 1163 raft_consensus.cc:357] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.899569 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 3bed3f04c60249aaa389c26788974f58 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:26.899626 1163 raft_consensus.cc:383] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.899678 1163 raft_consensus.cc:738] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.899924 1163 consensus_queue.cc:260] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.900775 1793 raft_consensus.cc:491] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.900885 1793 raft_consensus.cc:513] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.901356 1162 tablet_bootstrap.cc:654] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.901408 1793 leader_election.cc:290] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.901953 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:26.902297 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:26.902724 851 leader_election.cc:343] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:26.902969 850 leader_election.cc:343] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.903077 850 leader_election.cc:304] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:26.903388 1793 raft_consensus.cc:2747] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:26.906330 1163 ts_tablet_manager.cc:1428] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:26.906677 1163 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.910049 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9cacaaab2fce44ca9bb02a60b6edf5ff" } new_table_name: "default.Table_099213c8ceca49d6b4ba263ad5a52500" modify_external_catalogs: true
I20250625 07:52:26.916447 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 2 got 0
I20250625 07:52:26.919940 1160 ts_tablet_manager.cc:1918] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.921937 1160 log.cc:1199] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:26.922415 1160 ts_tablet_manager.cc:1939] T cbdbd9230210455ba615d51f1a98267b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:26.923695 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34a338005a7b44b9b0fba7a7096cfcb8 consensus state has no leader
I20250625 07:52:26.920121 1163 tablet_bootstrap.cc:654] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.925204 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2b0d5eec0a7841b1bba234b5e57b41f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.926246 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 799) took 1308 ms. Trace:
I20250625 07:52:26.926438 1160 rpcz_store.cc:276] 0625 07:52:25.617772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.617819 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:26.885079 (+1267260us) ts_tablet_manager.cc:1145] Deleting tablet cbdbd9230210455ba615d51f1a98267b
0625 07:52:26.885089 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:26.896217 (+ 11128us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.919833 (+ 23616us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:26.926219 (+ 6386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267008}
I20250625 07:52:26.926599 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5614a7608ac14d9dbf64878ba87a66de" } modify_external_catalogs: true
I20250625 07:52:26.926792 594 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 2 got 0
I20250625 07:52:26.926792 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5614a7608ac14d9dbf64878ba87a66de" } modify_external_catalogs: true
I20250625 07:52:26.927095 1160 tablet_replica.cc:331] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:26.927323 1160 raft_consensus.cc:2241] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:26.927476 1160 raft_consensus.cc:2270] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:26.928023 1160 ts_tablet_manager.cc:1905] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:26.928946 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet cbdbd9230210455ba615d51f1a98267b (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
I20250625 07:52:26.929665 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.930956 1166 ts_tablet_manager.cc:1918] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:26.931069 1166 log.cc:1199] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9418b81f010d4f369d3409e9364bc5fa
I20250625 07:52:26.931407 1166 ts_tablet_manager.cc:1939] T 9418b81f010d4f369d3409e9364bc5fa P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:26.935029 1733 raft_consensus.cc:491] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.935201 1733 raft_consensus.cc:513] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.935926 1733 leader_election.cc:290] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:26.936416 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:26.936998 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cb2aa6dac1f4443be9782aac8f855af" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:26.937621 986 leader_election.cc:304] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:26.937883 1733 raft_consensus.cc:2747] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:26.949792 1162 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:26.950047 1162 ts_tablet_manager.cc:1397] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.001s
I20250625 07:52:26.950882 1162 raft_consensus.cc:357] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.951166 1162 raft_consensus.cc:383] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.951284 1162 raft_consensus.cc:738] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.951597 1162 consensus_queue.cc:260] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.952898 1871 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:26.958927 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_634beccfc1a34257bd0bc04d1d020530" } modify_external_catalogs: true
I20250625 07:52:26.959264 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_634beccfc1a34257bd0bc04d1d020530" } modify_external_catalogs: true
I20250625 07:52:26.964114 1163 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:26.964265 1163 ts_tablet_manager.cc:1397] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.006s
I20250625 07:52:26.965744 1163 raft_consensus.cc:357] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.965905 1163 raft_consensus.cc:383] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:26.965948 1163 raft_consensus.cc:738] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:26.966164 1163 consensus_queue.cc:260] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:26.968432 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:26.944828 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9418b81f010d4f369d3409e9364bc5fa (table default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]) successfully deleted
W20250625 07:52:26.970743 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d605113ad8f246148825923e5bcdf00d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d605113ad8f246148825923e5bcdf00d already in progress: creating tablet
W20250625 07:52:26.971227 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c8c015c202544914b74f1e245b496f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8c015c202544914b74f1e245b496f82 already in progress: creating tablet
W20250625 07:52:26.970960 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b0d5eec0a7841b1bba234b5e57b41f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.972095 883 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:26.972576 1163 ts_tablet_manager.cc:1428] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:26.974941 1049 raft_consensus.cc:3058] T 9cb2aa6dac1f4443be9782aac8f855af P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:26.976673 1874 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:26.979000 1793 raft_consensus.cc:491] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:26.979128 1793 raft_consensus.cc:513] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:26.979780 1793 leader_election.cc:290] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:26.981467 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:26.982064 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:26.983964 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f2029eb1ac264cc9967ec89a919d5431" } modify_external_catalogs: true
I20250625 07:52:26.984175 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f2029eb1ac264cc9967ec89a919d5431" } modify_external_catalogs: true
I20250625 07:52:26.984331 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:26.984838 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7485d0b61b2242e688d4b47cebfcb9cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:26.976924 1873 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:26.986851 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7485d0b61b2242e688d4b47cebfcb9cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:26.987401 850 leader_election.cc:343] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:26.994688 1163 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:26.997213 1877 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:26.997481 1876 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:26.952091 1869 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:26.999068 1162 ts_tablet_manager.cc:1428] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250625 07:52:26.999588 1162 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.002967 594 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 6b3f2915b351407da08a9d6ba8ed402f (table default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555]). Expected version 2 got 0
I20250625 07:52:27.004801 1162 tablet_bootstrap.cc:654] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:26.952483 1870 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:26.974960 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:26.980667 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.006147 1166 tablet_replica.cc:331] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.006336 1166 raft_consensus.cc:2241] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.006371 1161 ts_tablet_manager.cc:1918] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.006475 1161 log.cc:1199] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8aa30ca0035048d3b60669994d402734
I20250625 07:52:27.006513 1166 raft_consensus.cc:2270] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:27.006619 851 leader_election.cc:343] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.006757 851 leader_election.cc:304] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.006851 1161 ts_tablet_manager.cc:1939] T 8aa30ca0035048d3b60669994d402734 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.007280 1166 ts_tablet_manager.cc:1905] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.007267 1793 raft_consensus.cc:2747] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.015381 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dc7e4a1ebfbe4239a253f502af55d599"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.016182 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc7e4a1ebfbe4239a253f502af55d599 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.016554 1166 ts_tablet_manager.cc:1918] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.016731 1166 log.cc:1199] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:27.017335 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0a285a3966f14deb8b22b7f4eacd7e71" } new_table_name: "default.Table_bbe3a0ab3ea64ad1ba4b2ec8bb99fff1" modify_external_catalogs: true
I20250625 07:52:27.017057 1166 ts_tablet_manager.cc:1939] T 2db9e8d0fefa47bfaeee462dba096b49 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.021689 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b1873a0532e24b77925f3c5b029dcda3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.022297 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2db9e8d0fefa47bfaeee462dba096b49 (table default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]) successfully deleted
I20250625 07:52:27.023062 1158 tablet_bootstrap.cc:492] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
W20250625 07:52:27.022424 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1873a0532e24b77925f3c5b029dcda3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.026377 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8aa30ca0035048d3b60669994d402734 (table default.Table_bcddc07c3a8a46449f20441de84992f6 [id=658ade951b8f490a9ed7d511dbc14fd1]) successfully deleted
I20250625 07:52:27.027297 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_1c27389c187843de9945c7a9e3d1bed4 [id=1ab8d615f8a84fdeba96d8266c1d4555] alter complete (version 2)
W20250625 07:52:27.030010 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2398a407b3154334a6a167946cf99ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2398a407b3154334a6a167946cf99ad7 already in progress: creating tablet
I20250625 07:52:27.030398 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_099213c8ceca49d6b4ba263ad5a52500" } new_table_name: "default.Table_3aeb5c70494240aaaad67f311ec1723b" modify_external_catalogs: true
I20250625 07:52:27.032610 1875 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 07:52:27.034106 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a0342575a2b34bf4b2390968e6acbe5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0342575a2b34bf4b2390968e6acbe5a already in progress: creating tablet
I20250625 07:52:27.038043 1162 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.038250 1162 ts_tablet_manager.cc:1397] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250625 07:52:27.039228 1162 raft_consensus.cc:357] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.040771 1162 raft_consensus.cc:383] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.041363 1162 raft_consensus.cc:738] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.045004 1163 tablet_bootstrap.cc:654] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.054318 1162 consensus_queue.cc:260] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.024655 1872 tablet.cc:1722] T 3eea1c257b584a029efa08922547b0cc P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:27.037604 1161 tablet_replica.cc:331] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.057081 1158 ts_tablet_manager.cc:1397] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.181s user 0.006s sys 0.001s
I20250625 07:52:27.057181 1161 raft_consensus.cc:2241] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.057348 1161 raft_consensus.cc:2270] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.057940 1158 raft_consensus.cc:357] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.058204 1158 raft_consensus.cc:383] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.058346 1158 raft_consensus.cc:738] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.058629 1158 consensus_queue.cc:260] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.059170 1158 ts_tablet_manager.cc:1428] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:27.057955 1161 ts_tablet_manager.cc:1905] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.061615 1162 ts_tablet_manager.cc:1428] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250625 07:52:27.061919 1162 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:27.068770 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fed43adfae594e3b91ed1012838c3317 consensus state has no leader
W20250625 07:52:27.070322 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34a338005a7b44b9b0fba7a7096cfcb8 consensus state has no leader
I20250625 07:52:27.071349 1162 tablet_bootstrap.cc:654] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.074030 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d4f6048941564dd399bc7458689264ab" } new_table_name: "default.Table_6984a582298e47918be6be498a5e24d9" modify_external_catalogs: true
I20250625 07:52:27.074285 1160 ts_tablet_manager.cc:1918] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.075595 1856 raft_consensus.cc:491] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.075734 1856 raft_consensus.cc:513] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.076068 1160 log.cc:1199] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/fcc3afc210a540058294121d2b83c66d
I20250625 07:52:27.076436 1856 leader_election.cc:290] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.080597 1160 ts_tablet_manager.cc:1939] T fcc3afc210a540058294121d2b83c66d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.080746 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:27.083374 912 raft_consensus.cc:2466] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:27.084101 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 800) took 1465 ms. Trace:
I20250625 07:52:27.084216 1160 rpcz_store.cc:276] 0625 07:52:25.618583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.618629 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:26.927040 (+1308411us) ts_tablet_manager.cc:1145] Deleting tablet fcc3afc210a540058294121d2b83c66d
0625 07:52:26.927053 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:26.951423 (+ 24370us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.074158 (+122735us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.084074 (+ 9916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308196}
I20250625 07:52:27.084782 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet fcc3afc210a540058294121d2b83c66d (table default.Table_bb831997b8cf4b99a7779a4a6b23a238 [id=8a01186fa596413fb75529d0d57a8d9d]) successfully deleted
I20250625 07:52:27.085172 1160 tablet_replica.cc:331] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.085397 1160 raft_consensus.cc:2241] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.085548 1160 raft_consensus.cc:2270] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.086184 1160 ts_tablet_manager.cc:1905] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.087246 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_75a5d24d45144619ab16307ddb8a5807" } new_table_name: "default.Table_0eb479026e15483586bb2df5333414d1" modify_external_catalogs: true
I20250625 07:52:27.088644 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cfb63fdad31e4cada3c0e785a501be1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.089651 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb63fdad31e4cada3c0e785a501be1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.090247 1163 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.090392 1163 ts_tablet_manager.cc:1397] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20250625 07:52:27.091285 1163 raft_consensus.cc:357] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.091499 1163 raft_consensus.cc:383] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.091558 1163 raft_consensus.cc:738] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.091781 1163 consensus_queue.cc:260] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.093384 1162 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.093560 1162 ts_tablet_manager.cc:1397] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250625 07:52:27.094306 1162 raft_consensus.cc:357] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.094529 1162 raft_consensus.cc:383] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.094630 1162 raft_consensus.cc:738] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.094861 1162 consensus_queue.cc:260] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.095381 1162 ts_tablet_manager.cc:1428] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:27.097157 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.097671 713 leader_election.cc:304] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:27.098057 713 leader_election.cc:343] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.098407 1850 raft_consensus.cc:2802] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:27.098536 1850 raft_consensus.cc:491] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:27.098996 1850 raft_consensus.cc:3058] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.102564 1850 raft_consensus.cc:513] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.103411 1850 leader_election.cc:290] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.103843 1163 ts_tablet_manager.cc:1428] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 07:52:27.104404 1162 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.105993 1162 tablet_bootstrap.cc:654] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.109082 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
W20250625 07:52:27.109828 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1796a3305cb94ccb9a0596643a736a96 consensus state has no leader
I20250625 07:52:27.110364 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d02d6575202148d0bad916a7cba0191d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.110447 1011 tablet_service.cc:1468] Processing CreateTablet for tablet fb78c180f0a6449ea3d152108cf26bf7 (DEFAULT_TABLE table=default.Table_b1909d0c3c834ca28b60d078f5c27129 [id=7a74cf1516a44921845710d35e650972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.110841 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb78c180f0a6449ea3d152108cf26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.110868 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d02d6575202148d0bad916a7cba0191d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.110729 883 tablet_service.cc:1468] Processing CreateTablet for tablet fb78c180f0a6449ea3d152108cf26bf7 (DEFAULT_TABLE table=default.Table_b1909d0c3c834ca28b60d078f5c27129 [id=7a74cf1516a44921845710d35e650972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.111781 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb78c180f0a6449ea3d152108cf26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.112565 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_9a7acbd7be614574a81e31a0f68bdbed [id=23322aa9321140c39272b5b2a102fb02] alter complete (version 3)
I20250625 07:52:27.113065 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_acf7718b3b9b41e2a0de6365460020fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.113536 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acf7718b3b9b41e2a0de6365460020fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:27.115214 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b160757deef94a008c500d961bbb19ab consensus state has no leader
I20250625 07:52:27.116021 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2c51d93cc45a4f19b57e86a55ed454a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.116618 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c51d93cc45a4f19b57e86a55ed454a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.127713 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 126e3c3c0b5e41599e7027708fefd7e7 (DEFAULT_TABLE table=default.Table_c0c5f7be7e344a1c9172a48d12556733 [id=5d1846ab019f4c44a9ef387649a7dc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.128257 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e3c3c0b5e41599e7027708fefd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.133491 1158 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.139119 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.139326 890 tablet_service.cc:1468] Processing CreateTablet for tablet 032a37d843dc4daf8f9cce6843a3db6d (DEFAULT_TABLE table=default.Table_15eb4e6d76ae4533aeee6d180c721b13 [id=60fc09b4e0f44775a4dcf81554794b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.139534 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6881336f1ff44e3cb7d1bc8937e503a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.139843 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032a37d843dc4daf8f9cce6843a3db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.140038 879 tablet_service.cc:1468] Processing CreateTablet for tablet 52c1f7cdd88d423788d7d30f15318786 (DEFAULT_TABLE table=default.Table_8fcfcfd3d18a44818c59195ab07e9f57 [id=b7c2563be5c4460fba9e8e458546bfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.140334 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c1f7cdd88d423788d7d30f15318786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.136293 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 032a37d843dc4daf8f9cce6843a3db6d (DEFAULT_TABLE table=default.Table_15eb4e6d76ae4533aeee6d180c721b13 [id=60fc09b4e0f44775a4dcf81554794b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.140820 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032a37d843dc4daf8f9cce6843a3db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.140973 893 tablet_service.cc:1468] Processing CreateTablet for tablet 833cb1c75c5e429dac3428784007ab40 (DEFAULT_TABLE table=default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62 [id=86a6e50d7f414d06a588a1a4c6f64882]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.141269 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cb1c75c5e429dac3428784007ab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.136487 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 52c1f7cdd88d423788d7d30f15318786 (DEFAULT_TABLE table=default.Table_8fcfcfd3d18a44818c59195ab07e9f57 [id=b7c2563be5c4460fba9e8e458546bfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.144881 1793 raft_consensus.cc:491] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.144991 1793 raft_consensus.cc:513] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.145097 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c1f7cdd88d423788d7d30f15318786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.145624 1793 leader_election.cc:290] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.146306 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1898ff5b7626478bb997f7a433477498" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:27.146867 851 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.136649 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 5e788b913736432c9ef000940fa58f24 (DEFAULT_TABLE table=default.Table_5aa94b384aea4fedbba2d3431d7ee5f8 [id=1aa6f71dbc394becbc5d838194e78e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.149102 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e788b913736432c9ef000940fa58f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.150406 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:27.150985 755 tablet_service.cc:1468] Processing CreateTablet for tablet fb78c180f0a6449ea3d152108cf26bf7 (DEFAULT_TABLE table=default.Table_b1909d0c3c834ca28b60d078f5c27129 [id=7a74cf1516a44921845710d35e650972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.151163 753 tablet_service.cc:1468] Processing CreateTablet for tablet d784930801e14b1a91d893df53ab71a7 (DEFAULT_TABLE table=default.Table_d715361336db4dcab94ab01d4f00ce32 [id=aca081d1bd6344ac922c06878d1bf982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.151355 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb78c180f0a6449ea3d152108cf26bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.151554 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d784930801e14b1a91d893df53ab71a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.151821 747 tablet_service.cc:1468] Processing CreateTablet for tablet 126e3c3c0b5e41599e7027708fefd7e7 (DEFAULT_TABLE table=default.Table_c0c5f7be7e344a1c9172a48d12556733 [id=5d1846ab019f4c44a9ef387649a7dc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.152109 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e3c3c0b5e41599e7027708fefd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.153035 888 tablet_service.cc:1468] Processing CreateTablet for tablet d784930801e14b1a91d893df53ab71a7 (DEFAULT_TABLE table=default.Table_d715361336db4dcab94ab01d4f00ce32 [id=aca081d1bd6344ac922c06878d1bf982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.153405 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d784930801e14b1a91d893df53ab71a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.153613 749 tablet_service.cc:1468] Processing CreateTablet for tablet 833cb1c75c5e429dac3428784007ab40 (DEFAULT_TABLE table=default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62 [id=86a6e50d7f414d06a588a1a4c6f64882]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.153899 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cb1c75c5e429dac3428784007ab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.136651 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 833cb1c75c5e429dac3428784007ab40 (DEFAULT_TABLE table=default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62 [id=86a6e50d7f414d06a588a1a4c6f64882]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.156973 889 tablet_service.cc:1468] Processing CreateTablet for tablet 75e882e974da4e929c1301b56fbf66fa (DEFAULT_TABLE table=default.Table_3c26e45d95314578aa155e978ffd1096 [id=aa3db86b38cf42cc8c0a0f5114b801dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.157166 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cb1c75c5e429dac3428784007ab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.157205 876 tablet_service.cc:1468] Processing CreateTablet for tablet a05967c96d5749f8a02f232ace35be7c (DEFAULT_TABLE table=default.Table_3617e2588af44bbea69c7b4181f219e9 [id=a9c2ee51808c4869813fffef2d195f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.157334 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e882e974da4e929c1301b56fbf66fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.157482 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05967c96d5749f8a02f232ace35be7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.156988 892 tablet_service.cc:1468] Processing CreateTablet for tablet 78f0921af91140c6989de2306326c28f (DEFAULT_TABLE table=default.Table_c6fe509bff964fc783b30da1badc3431 [id=b03c78f4510940468acb323eb3044d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.160856 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f0921af91140c6989de2306326c28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.136869 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d784930801e14b1a91d893df53ab71a7 (DEFAULT_TABLE table=default.Table_d715361336db4dcab94ab01d4f00ce32 [id=aca081d1bd6344ac922c06878d1bf982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.137777 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.160972 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6a80efc0779a4df6a1b65f74f2745661 (DEFAULT_TABLE table=default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb [id=7c0443bff4a4416faf03c15e5d151ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.137955 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 78f0921af91140c6989de2306326c28f (DEFAULT_TABLE table=default.Table_c6fe509bff964fc783b30da1badc3431 [id=b03c78f4510940468acb323eb3044d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.161260 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a80efc0779a4df6a1b65f74f2745661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.137046 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 75e882e974da4e929c1301b56fbf66fa (DEFAULT_TABLE table=default.Table_3c26e45d95314578aa155e978ffd1096 [id=aa3db86b38cf42cc8c0a0f5114b801dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.137226 1013 tablet_service.cc:1468] Processing CreateTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.137609 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a05967c96d5749f8a02f232ace35be7c (DEFAULT_TABLE table=default.Table_3617e2588af44bbea69c7b4181f219e9 [id=a9c2ee51808c4869813fffef2d195f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.137429 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6a80efc0779a4df6a1b65f74f2745661 (DEFAULT_TABLE table=default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb [id=7c0443bff4a4416faf03c15e5d151ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.161141 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d784930801e14b1a91d893df53ab71a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.164989 895 tablet_service.cc:1468] Processing CreateTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.165362 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2f623e8c9742c28a97ce3a4b16b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.168193 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6881336f1ff44e3cb7d1bc8937e503a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.153411 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cffde34fbba34e44b67985d3480666c3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:27.169449 912 raft_consensus.cc:3058] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.172433 912 raft_consensus.cc:2466] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:27.172818 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2f623e8c9742c28a97ce3a4b16b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.176254 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05967c96d5749f8a02f232ace35be7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.180809 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a80efc0779a4df6a1b65f74f2745661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.185575 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e882e974da4e929c1301b56fbf66fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.188930 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f0921af91140c6989de2306326c28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.198235 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1898ff5b7626478bb997f7a433477498" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.198338 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:27.199194 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fdcd25178a124f4fbe5c676be9adb5fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.200209 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdcd25178a124f4fbe5c676be9adb5fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.200596 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f04016cda7f746d49e30be3b1c9c18b6
I20250625 07:52:27.201752 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fc290bab1df44405ab85e0058a8bc848" } new_table_name: "default.Table_8e925fa4bed3456fb11fa79d527a8a42" modify_external_catalogs: true
W20250625 07:52:27.202385 850 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.202510 850 leader_election.cc:304] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.202769 1793 raft_consensus.cc:2747] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.203166 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 52c1f7cdd88d423788d7d30f15318786
I20250625 07:52:27.203174 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.203358 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.203385 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.203423 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_12bc5fb366f644f1a5566b904a5c5ba9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.203775 895 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:27.203924 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12bc5fb366f644f1a5566b904a5c5ba9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.205044 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5b0d357cefb84a239d5be73857b9097b
I20250625 07:52:27.205785 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d357cefb84a239d5be73857b9097b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.205757 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2b2ab9e4f189492dbfec4ec5c79a1630"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.206066 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d357cefb84a239d5be73857b9097b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:27.206472 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b2ab9e4f189492dbfec4ec5c79a1630 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.208110 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_99eddf6f5e75402fb97ac6cc3375e188"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.208200 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c1f7cdd88d423788d7d30f15318786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:27.208562 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99eddf6f5e75402fb97ac6cc3375e188 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.209569 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c1f7cdd88d423788d7d30f15318786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.209906 1793 raft_consensus.cc:491] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.210075 1793 raft_consensus.cc:513] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.210749 1793 leader_election.cc:290] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.211426 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7b755d95b14ecea7a691003455f381" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.211580 753 tablet_service.cc:1468] Processing CreateTablet for tablet 75e882e974da4e929c1301b56fbf66fa (DEFAULT_TABLE table=default.Table_3c26e45d95314578aa155e978ffd1096 [id=aa3db86b38cf42cc8c0a0f5114b801dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.211660 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6a80efc0779a4df6a1b65f74f2745661 (DEFAULT_TABLE table=default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb [id=7c0443bff4a4416faf03c15e5d151ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.211625 749 tablet_service.cc:1468] Processing CreateTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.212013 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e882e974da4e929c1301b56fbf66fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.212030 851 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.212240 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2f623e8c9742c28a97ce3a4b16b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.212373 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.212486 743 tablet_service.cc:1468] Processing CreateTablet for tablet 52c1f7cdd88d423788d7d30f15318786 (DEFAULT_TABLE table=default.Table_8fcfcfd3d18a44818c59195ab07e9f57 [id=b7c2563be5c4460fba9e8e458546bfe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.212571 742 tablet_service.cc:1468] Processing CreateTablet for tablet a05967c96d5749f8a02f232ace35be7c (DEFAULT_TABLE table=default.Table_3617e2588af44bbea69c7b4181f219e9 [id=a9c2ee51808c4869813fffef2d195f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.212675 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6881336f1ff44e3cb7d1bc8937e503a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.212850 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05967c96d5749f8a02f232ace35be7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.212996 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c1f7cdd88d423788d7d30f15318786. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.213142 744 tablet_service.cc:1468] Processing CreateTablet for tablet 032a37d843dc4daf8f9cce6843a3db6d (DEFAULT_TABLE table=default.Table_15eb4e6d76ae4533aeee6d180c721b13 [id=60fc09b4e0f44775a4dcf81554794b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.213430 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 032a37d843dc4daf8f9cce6843a3db6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.216938 754 tablet_service.cc:1468] Processing CreateTablet for tablet 78f0921af91140c6989de2306326c28f (DEFAULT_TABLE table=default.Table_c6fe509bff964fc783b30da1badc3431 [id=b03c78f4510940468acb323eb3044d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.217245 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f0921af91140c6989de2306326c28f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.218815 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5e788b913736432c9ef000940fa58f24 (DEFAULT_TABLE table=default.Table_5aa94b384aea4fedbba2d3431d7ee5f8 [id=1aa6f71dbc394becbc5d838194e78e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.219096 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e788b913736432c9ef000940fa58f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.220795 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a80efc0779a4df6a1b65f74f2745661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.224620 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.224733 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d357cefb84a239d5be73857b9097b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.225121 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c1f7cdd88d423788d7d30f15318786 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.226001 713 leader_election.cc:304] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:27.226282 1856 raft_consensus.cc:2802] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:27.226395 1856 raft_consensus.cc:695] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:27.226613 1856 consensus_queue.cc:237] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.226922 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7b755d95b14ecea7a691003455f381" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.228192 850 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.228339 850 leader_election.cc:304] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.228821 1793 raft_consensus.cc:2747] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.233667 895 tablet_service.cc:1468] Processing CreateTablet for tablet 5e788b913736432c9ef000940fa58f24 (DEFAULT_TABLE table=default.Table_5aa94b384aea4fedbba2d3431d7ee5f8 [id=1aa6f71dbc394becbc5d838194e78e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.234097 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e788b913736432c9ef000940fa58f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.235250 888 tablet_service.cc:1468] Processing CreateTablet for tablet 157590e34b8442b28e01f5c1c4b7d905 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.235611 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157590e34b8442b28e01f5c1c4b7d905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.243835 883 tablet_service.cc:1468] Processing CreateTablet for tablet 126e3c3c0b5e41599e7027708fefd7e7 (DEFAULT_TABLE table=default.Table_c0c5f7be7e344a1c9172a48d12556733 [id=5d1846ab019f4c44a9ef387649a7dc5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.244387 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e3c3c0b5e41599e7027708fefd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.246352 1161 ts_tablet_manager.cc:1918] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.246480 1161 log.cc:1199] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/5f62b03db94d4ee3935866b5dd8bd53f
I20250625 07:52:27.246894 1793 raft_consensus.cc:491] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.247011 1793 raft_consensus.cc:513] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.247589 1793 leader_election.cc:290] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:27.233767 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f04016cda7f746d49e30be3b1c9c18b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f04016cda7f746d49e30be3b1c9c18b6 already in progress: creating tablet
W20250625 07:52:27.248162 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5b0d357cefb84a239d5be73857b9097b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0d357cefb84a239d5be73857b9097b already in progress: creating tablet
I20250625 07:52:27.248358 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0735d576f947d395bee8d1b865f2fd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:27.248385 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 52c1f7cdd88d423788d7d30f15318786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c1f7cdd88d423788d7d30f15318786 already in progress: creating tablet
W20250625 07:52:27.249017 851 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.249356 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0735d576f947d395bee8d1b865f2fd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.249754 1162 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.249926 1162 ts_tablet_manager.cc:1397] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.146s user 0.009s sys 0.019s
I20250625 07:52:27.249997 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 157590e34b8442b28e01f5c1c4b7d905 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.250535 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157590e34b8442b28e01f5c1c4b7d905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.250777 1162 raft_consensus.cc:357] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.250929 1162 raft_consensus.cc:383] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.250975 1162 raft_consensus.cc:738] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.251066 1733 raft_consensus.cc:491] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.251163 1162 consensus_queue.cc:260] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.251168 1733 raft_consensus.cc:513] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.251593 1162 ts_tablet_manager.cc:1428] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:27.251730 1733 leader_election.cc:290] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:27.251854 1162 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:27.252671 986 leader_election.cc:343] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: de9ccb0a2d3846228acbc2769786a059
W20250625 07:52:27.253518 985 leader_election.cc:343] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: de9ccb0a2d3846228acbc2769786a059
I20250625 07:52:27.253643 985 leader_election.cc:304] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:27.254452 1733 raft_consensus.cc:2747] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.257305 1162 tablet_bootstrap.cc:654] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:27.258832 850 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.259006 850 leader_election.cc:304] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.259423 1793 raft_consensus.cc:2747] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.135691 1158 tablet_bootstrap.cc:654] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.263521 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 9bd26bc5a4024f1b9919871a8a06b628 (DEFAULT_TABLE table=default.Table_3f977dd0493949a8afda3f376504f89b [id=e62d2bbd5bd843f1a65d4e7017e56150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.263875 1160 ts_tablet_manager.cc:1918] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.264065 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd26bc5a4024f1b9919871a8a06b628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.265439 1162 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.266000 1160 log.cc:1199] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0f566cc9cdd447798a5cb408ef9cd35a
I20250625 07:52:27.268860 1162 ts_tablet_manager.cc:1397] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250625 07:52:27.269039 1163 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.270793 1160 ts_tablet_manager.cc:1939] T 0f566cc9cdd447798a5cb408ef9cd35a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.271700 1163 tablet_bootstrap.cc:654] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.272851 1162 raft_consensus.cc:357] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.273641 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 801) took 1654 ms. Trace:
I20250625 07:52:27.273799 1162 raft_consensus.cc:383] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.274025 1160 rpcz_store.cc:276] 0625 07:52:25.619219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.619262 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:27.085113 (+1465851us) ts_tablet_manager.cc:1145] Deleting tablet 0f566cc9cdd447798a5cb408ef9cd35a
0625 07:52:27.085125 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.246469 (+161344us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.263772 (+ 17303us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.273617 (+ 9845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465620}
I20250625 07:52:27.274338 1162 raft_consensus.cc:738] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.275645 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0f566cc9cdd447798a5cb408ef9cd35a (table default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]) successfully deleted
I20250625 07:52:27.276151 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 812) took 1517 ms. Trace:
I20250625 07:52:27.275075 1162 consensus_queue.cc:260] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.281199 1162 ts_tablet_manager.cc:1428] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 07:52:27.277029 890 tablet_service.cc:1468] Processing CreateTablet for tablet 9bd26bc5a4024f1b9919871a8a06b628 (DEFAULT_TABLE table=default.Table_3f977dd0493949a8afda3f376504f89b [id=e62d2bbd5bd843f1a65d4e7017e56150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.279134 1793 raft_consensus.cc:491] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.281531 1793 raft_consensus.cc:513] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.281764 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd26bc5a4024f1b9919871a8a06b628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.282227 1793 leader_election.cc:290] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.282598 1162 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.283013 1158 tablet_bootstrap.cc:492] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.282600 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ec29e4294864fbabdff5a04dcf5b163" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.283221 1158 ts_tablet_manager.cc:1397] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.002s
I20250625 07:52:27.282938 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ec29e4294864fbabdff5a04dcf5b163" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.283900 851 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.283941 1158 raft_consensus.cc:357] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.284369 1162 tablet_bootstrap.cc:654] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:27.284381 850 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.284513 850 leader_election.cc:304] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.284759 1793 raft_consensus.cc:2747] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.285156 1158 raft_consensus.cc:383] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.285418 1158 raft_consensus.cc:738] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.289037 1161 ts_tablet_manager.cc:1939] T 5f62b03db94d4ee3935866b5dd8bd53f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.290437 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 5f62b03db94d4ee3935866b5dd8bd53f (table default.Table_b9d2937a36aa4a7daf58b3c859ab9c94 [id=d3319c9f4d6d457a8b8cdab1e0b89d23]) successfully deleted
I20250625 07:52:27.290777 1161 tablet_replica.cc:331] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.291000 1161 raft_consensus.cc:2241] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.291200 1161 raft_consensus.cc:2270] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.291742 1161 ts_tablet_manager.cc:1905] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.292519 1162 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.292829 1733 raft_consensus.cc:491] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.292948 1733 raft_consensus.cc:513] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.293237 1162 ts_tablet_manager.cc:1397] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250625 07:52:27.293494 1733 leader_election.cc:290] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:27.294509 986 leader_election.cc:343] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 49697a10cdac4c8ebb69f10f4c3437ec
W20250625 07:52:27.294710 985 leader_election.cc:343] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 49697a10cdac4c8ebb69f10f4c3437ec
I20250625 07:52:27.294786 985 leader_election.cc:304] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:27.295112 1733 raft_consensus.cc:2747] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:27.288761 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 62786.9us
W20250625 07:52:27.295396 713 leader_election.cc:343] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.297330 755 tablet_service.cc:1468] Processing CreateTablet for tablet 157590e34b8442b28e01f5c1c4b7d905 (DEFAULT_TABLE table=default.Table_26b14eff630948cb807b96dfd6e192c7 [id=184e52d735294771a1f5bfcd2813783d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.297765 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157590e34b8442b28e01f5c1c4b7d905. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.299149 1162 raft_consensus.cc:357] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.299585 1162 raft_consensus.cc:383] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.299742 1162 raft_consensus.cc:738] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.300001 895 tablet_service.cc:1468] Processing CreateTablet for tablet 9a1e2434d6f04ac099d67bdc3ebdf335 (DEFAULT_TABLE table=default.Table_7b42526a3fb543b894d40b54df370f55 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.300153 1162 consensus_queue.cc:260] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.300411 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1e2434d6f04ac099d67bdc3ebdf335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.300504 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9a1e2434d6f04ac099d67bdc3ebdf335 (DEFAULT_TABLE table=default.Table_7b42526a3fb543b894d40b54df370f55 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.300855 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1e2434d6f04ac099d67bdc3ebdf335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.303123 1162 ts_tablet_manager.cc:1428] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 07:52:27.303378 1162 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.304729 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2c85bcbaa41c46b6b0546e259a44e6f4 (DEFAULT_TABLE table=default.Table_e40dd0454868478dadc2bc891d78f59c [id=d8b0f7dd1ed84fc3a655310740345791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.304929 1162 tablet_bootstrap.cc:654] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.305814 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 2c85bcbaa41c46b6b0546e259a44e6f4 (DEFAULT_TABLE table=default.Table_e40dd0454868478dadc2bc891d78f59c [id=d8b0f7dd1ed84fc3a655310740345791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.306368 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c85bcbaa41c46b6b0546e259a44e6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.311450 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3 (DEFAULT_TABLE table=default.Table_d907bad8a74a495a9bb4156ce049cd5d [id=ba04ebd5d9fc4ecc8476fc5dbfa6e4d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.311836 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.312183 895 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3 (DEFAULT_TABLE table=default.Table_d907bad8a74a495a9bb4156ce049cd5d [id=ba04ebd5d9fc4ecc8476fc5dbfa6e4d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.312572 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.312816 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 581ac5caf07f4280801d5ec8b8c90b76 (DEFAULT_TABLE table=default.Table_b6ed2b0b8a3c43d2a87a016f9d8464f8 [id=9e26b3f368c64da9bb85c76665d78070]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.313122 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 581ac5caf07f4280801d5ec8b8c90b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.305078 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c85bcbaa41c46b6b0546e259a44e6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.319533 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3ea1daff2b4d15874c00ff075be8be (DEFAULT_TABLE table=default.Table_d3a57960634c41bd8b2bb81061f1a4ab [id=1dffa72e03de4ff289c169ae339eeed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.320196 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3ea1daff2b4d15874c00ff075be8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.305399 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9bd26bc5a4024f1b9919871a8a06b628 (DEFAULT_TABLE table=default.Table_3f977dd0493949a8afda3f376504f89b [id=e62d2bbd5bd843f1a65d4e7017e56150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.321059 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd26bc5a4024f1b9919871a8a06b628. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.288970 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.325320 888 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3ea1daff2b4d15874c00ff075be8be (DEFAULT_TABLE table=default.Table_d3a57960634c41bd8b2bb81061f1a4ab [id=1dffa72e03de4ff289c169ae339eeed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.325765 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3ea1daff2b4d15874c00ff075be8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.328812 1158 consensus_queue.cc:260] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.329907 1158 ts_tablet_manager.cc:1428] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20250625 07:52:27.330859 1158 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.332226 1158 tablet_bootstrap.cc:654] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.281424 1160 rpcz_store.cc:276] 0625 07:52:25.759191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.759263 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:27.276039 (+1516776us) ts_tablet_manager.cc:1145] Deleting tablet 2afbfb6fdb164ae2917ebf1bbb6433fe
0625 07:52:27.276047 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.276137 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516477}
W20250625 07:52:27.340443 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2afbfb6fdb164ae2917ebf1bbb6433fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2afbfb6fdb164ae2917ebf1bbb6433fe already in progress: creating tablet
I20250625 07:52:27.340934 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 813) took 1580 ms. Trace:
I20250625 07:52:27.341106 1160 rpcz_store.cc:276] 0625 07:52:25.760571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.760627 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:27.340811 (+1580184us) ts_tablet_manager.cc:1145] Deleting tablet 6b39c48d9948400a8f36e389cff0020a
0625 07:52:27.340823 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.340918 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1579938}
W20250625 07:52:27.341609 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6b39c48d9948400a8f36e389cff0020a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b39c48d9948400a8f36e389cff0020a already in progress: creating tablet
I20250625 07:52:27.342172 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 828) took 1356 ms. Trace:
I20250625 07:52:27.342286 1160 rpcz_store.cc:276] 0625 07:52:25.985548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.985597 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:27.342048 (+1356451us) ts_tablet_manager.cc:1145] Deleting tablet 47b4112b7e674518bcc1ea748370fd85
0625 07:52:27.342057 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.342156 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1332347}
W20250625 07:52:27.342707 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 47b4112b7e674518bcc1ea748370fd85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47b4112b7e674518bcc1ea748370fd85 already in progress: creating tablet
I20250625 07:52:27.343057 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 829) took 1357 ms. Trace:
I20250625 07:52:27.343159 1160 rpcz_store.cc:276] 0625 07:52:25.985931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.985993 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:27.342955 (+1356962us) ts_tablet_manager.cc:1145] Deleting tablet cac7d20327e14f90af25acb58d724b68
0625 07:52:27.342963 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.343043 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333183}
W20250625 07:52:27.343582 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet cac7d20327e14f90af25acb58d724b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cac7d20327e14f90af25acb58d724b68 already in progress: creating tablet
I20250625 07:52:27.343930 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 832) took 1352 ms. Trace:
I20250625 07:52:27.344041 1160 rpcz_store.cc:276] 0625 07:52:25.991162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.991214 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:27.343838 (+1352624us) ts_tablet_manager.cc:1145] Deleting tablet 0aeb7077591f466eb77e3096f6b1d74b
0625 07:52:27.343846 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.343917 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333889}
W20250625 07:52:27.344447 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0aeb7077591f466eb77e3096f6b1d74b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0aeb7077591f466eb77e3096f6b1d74b already in progress: creating tablet
I20250625 07:52:27.352663 1163 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.352829 1163 ts_tablet_manager.cc:1397] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.001s
W20250625 07:52:27.353116 584 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:43317, user_credentials={real_user=slave}} blocked reactor thread for 70511.5us
I20250625 07:52:27.353646 1163 raft_consensus.cc:357] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.353895 1163 raft_consensus.cc:383] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.353989 1163 raft_consensus.cc:738] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.354256 1163 consensus_queue.cc:260] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:27.354688 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4167282133f94bfb9870bd08fc6594b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4167282133f94bfb9870bd08fc6594b4 already in progress: creating tablet
I20250625 07:52:27.361337 895 tablet_service.cc:1468] Processing CreateTablet for tablet dd68bf68877346d1a46ac4511975a3e2 (DEFAULT_TABLE table=default.Table_9e878ae7dd0c46c685ce00f99d31a339 [id=ee2365ed252f4136978c9d31eedf9473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.361763 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68bf68877346d1a46ac4511975a3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.362784 889 tablet_service.cc:1468] Processing CreateTablet for tablet d38786a3d3a9450bb6efde1675302683 (DEFAULT_TABLE table=default.Table_2f5b39a2075a487d8cbe4d4c9319f779 [id=0696d675b4ba4fad8cb6539b79d5519c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.363129 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38786a3d3a9450bb6efde1675302683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.363384 883 tablet_service.cc:1468] Processing CreateTablet for tablet 2c85bcbaa41c46b6b0546e259a44e6f4 (DEFAULT_TABLE table=default.Table_e40dd0454868478dadc2bc891d78f59c [id=d8b0f7dd1ed84fc3a655310740345791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.363896 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c85bcbaa41c46b6b0546e259a44e6f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.368674 1163 ts_tablet_manager.cc:1428] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250625 07:52:27.372274 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3 (DEFAULT_TABLE table=default.Table_d907bad8a74a495a9bb4156ce049cd5d [id=ba04ebd5d9fc4ecc8476fc5dbfa6e4d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.372615 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4c3ea1daff2b4d15874c00ff075be8be (DEFAULT_TABLE table=default.Table_d3a57960634c41bd8b2bb81061f1a4ab [id=1dffa72e03de4ff289c169ae339eeed0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.372736 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0d0c87d5ca4db586c27d6e13d65fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.372982 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c3ea1daff2b4d15874c00ff075be8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.373827 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:27.374377 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 5de0c5bf60b24808af124c95833d9c25 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:27.374835 746 tablet_service.cc:1468] Processing CreateTablet for tablet dd68bf68877346d1a46ac4511975a3e2 (DEFAULT_TABLE table=default.Table_9e878ae7dd0c46c685ce00f99d31a339 [id=ee2365ed252f4136978c9d31eedf9473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.375180 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68bf68877346d1a46ac4511975a3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.375372 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2185371a6bd1488b88a8038243706d02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.375629 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b consensus state has no leader
W20250625 07:52:27.375928 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2185371a6bd1488b88a8038243706d02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.376053 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_15c9f1d9e9e842ef8ce4b125b9350af3" } new_table_name: "default.Table_cf2182c3809041cdaf5afe8185107e0f" modify_external_catalogs: true
I20250625 07:52:27.376245 755 tablet_service.cc:1468] Processing CreateTablet for tablet 581ac5caf07f4280801d5ec8b8c90b76 (DEFAULT_TABLE table=default.Table_b6ed2b0b8a3c43d2a87a016f9d8464f8 [id=9e26b3f368c64da9bb85c76665d78070]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.376605 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 581ac5caf07f4280801d5ec8b8c90b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.376657 749 tablet_service.cc:1468] Processing CreateTablet for tablet d38786a3d3a9450bb6efde1675302683 (DEFAULT_TABLE table=default.Table_2f5b39a2075a487d8cbe4d4c9319f779 [id=0696d675b4ba4fad8cb6539b79d5519c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.377095 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_2acd2b198c4743d5a1ed1b69991d0450 [id=3509d763da554384ba8b5ecefdbc5fab] alter complete (version 3)
I20250625 07:52:27.377331 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38786a3d3a9450bb6efde1675302683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.377490 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d218655737a6439da5c24f4643f48c76" } new_table_name: "default.Table_744b58275931414c9463ddc96dfe14aa" modify_external_catalogs: true
I20250625 07:52:27.377934 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8d4a225cda044dba96006acca0260845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.378397 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d4a225cda044dba96006acca0260845 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.380928 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9a1e2434d6f04ac099d67bdc3ebdf335 (DEFAULT_TABLE table=default.Table_7b42526a3fb543b894d40b54df370f55 [id=661bf2c60ea2451881cdd4508608b64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.381278 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1e2434d6f04ac099d67bdc3ebdf335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.381803 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ccd929ffd026433c9abd3851bb8e1004
I20250625 07:52:27.382527 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ccd929ffd026433c9abd3851bb8e1004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.382570 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b2cb8c1f912f43bfad73814acbecbc34" } new_table_name: "default.Table_4a950eac635448a1ac3562d75314e352" modify_external_catalogs: true
I20250625 07:52:27.382923 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet de9ccb0a2d3846228acbc2769786a059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.383183 889 tablet_service.cc:1515] Processing DeleteTablet for tablet ccd929ffd026433c9abd3851bb8e1004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.384574 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca22b663c92246f69b5beb7072264679" } new_table_name: "default.Table_ff871a8a9993429da4d087d0362c3ced" modify_external_catalogs: true
I20250625 07:52:27.386509 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.388283 1163 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.385721 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a559f40e04b9464ab39d21bf2fdd6ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.389521 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a559f40e04b9464ab39d21bf2fdd6ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.390090 1163 tablet_bootstrap.cc:654] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.391009 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d38786a3d3a9450bb6efde1675302683 (DEFAULT_TABLE table=default.Table_2f5b39a2075a487d8cbe4d4c9319f779 [id=0696d675b4ba4fad8cb6539b79d5519c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.391470 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d38786a3d3a9450bb6efde1675302683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.398794 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ccd929ffd026433c9abd3851bb8e1004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.399106 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.399598 1166 tablet_replica.cc:331] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.399780 1166 raft_consensus.cc:2241] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.399945 1166 raft_consensus.cc:2270] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.401024 1166 ts_tablet_manager.cc:1905] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.401046 890 tablet_service.cc:1468] Processing CreateTablet for tablet 581ac5caf07f4280801d5ec8b8c90b76 (DEFAULT_TABLE table=default.Table_b6ed2b0b8a3c43d2a87a016f9d8464f8 [id=9e26b3f368c64da9bb85c76665d78070]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:27.405364 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 80135.3us
I20250625 07:52:27.409503 1166 ts_tablet_manager.cc:1918] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.409713 1166 log.cc:1199] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:27.410207 1166 ts_tablet_manager.cc:1939] T 960ded22e2eb42d9a3494c020a4ea354 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.413082 1793 raft_consensus.cc:491] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.413137 895 tablet_service.cc:1468] Processing CreateTablet for tablet d0b056492dd84dbf9b4e7635e6751fef (DEFAULT_TABLE table=default.Table_d14f9617cfa34c00ae651e78dc5be263 [id=b7b4736a3aec4a48a20b33a0c719d55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.413226 1793 raft_consensus.cc:513] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.413808 1793 leader_election.cc:290] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.414229 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6c3be49be14ecbb3c26ef107ede015" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.414474 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6c3be49be14ecbb3c26ef107ede015" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.414753 851 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:27.415108 850 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.415212 850 leader_election.cc:304] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.415448 1793 raft_consensus.cc:2747] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.416154 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 132a3c4323834af78fecf29adbbbdd9d (DEFAULT_TABLE table=default.Table_1ef2001aff1f47f28501fb568a189e30 [id=e7fc3ccf3e3c41e4ae1c9efe872f72a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.416807 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a3c4323834af78fecf29adbbbdd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.417207 1014 tablet_service.cc:1468] Processing CreateTablet for tablet d0b056492dd84dbf9b4e7635e6751fef (DEFAULT_TABLE table=default.Table_d14f9617cfa34c00ae651e78dc5be263 [id=b7b4736a3aec4a48a20b33a0c719d55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.417644 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b056492dd84dbf9b4e7635e6751fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.420933 1011 tablet_service.cc:1468] Processing CreateTablet for tablet dd68bf68877346d1a46ac4511975a3e2 (DEFAULT_TABLE table=default.Table_9e878ae7dd0c46c685ce00f99d31a339 [id=ee2365ed252f4136978c9d31eedf9473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.421420 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd68bf68877346d1a46ac4511975a3e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.429133 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ccd929ffd026433c9abd3851bb8e1004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccd929ffd026433c9abd3851bb8e1004 already in progress: creating tablet
I20250625 07:52:27.431854 746 tablet_service.cc:1468] Processing CreateTablet for tablet d0b056492dd84dbf9b4e7635e6751fef (DEFAULT_TABLE table=default.Table_d14f9617cfa34c00ae651e78dc5be263 [id=b7b4736a3aec4a48a20b33a0c719d55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.432291 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b056492dd84dbf9b4e7635e6751fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.435797 749 tablet_service.cc:1468] Processing CreateTablet for tablet 132a3c4323834af78fecf29adbbbdd9d (DEFAULT_TABLE table=default.Table_1ef2001aff1f47f28501fb568a189e30 [id=e7fc3ccf3e3c41e4ae1c9efe872f72a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.436137 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a3c4323834af78fecf29adbbbdd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.439597 1161 ts_tablet_manager.cc:1918] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.439708 1161 log.cc:1199] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1e6c2988baa14cbc8c878f79e942880d
I20250625 07:52:27.439997 1161 ts_tablet_manager.cc:1939] T 1e6c2988baa14cbc8c878f79e942880d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.440284 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 815) took 1019 ms. Trace:
I20250625 07:52:27.440364 1161 rpcz_store.cc:276] 0625 07:52:26.420390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.420445 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:27.290731 (+870286us) ts_tablet_manager.cc:1145] Deleting tablet 1e6c2988baa14cbc8c878f79e942880d
0625 07:52:27.290742 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.357905 (+ 67163us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.439503 (+ 81598us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.440269 (+ 766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":870131}
I20250625 07:52:27.440946 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 1e6c2988baa14cbc8c878f79e942880d (table default.Table_f54f59a06d1e4cad83c44dd16452812d [id=1f266351dd3347cdab65609b863f1a7e]) successfully deleted
I20250625 07:52:27.441973 1793 raft_consensus.cc:491] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.442178 1733 raft_consensus.cc:491] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.442162 1793 raft_consensus.cc:513] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.442332 1733 raft_consensus.cc:513] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.442967 1793 leader_election.cc:290] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.443003 1733 leader_election.cc:290] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:27.444144 986 leader_election.cc:343] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:27.444115 879 tablet_service.cc:1468] Processing CreateTablet for tablet 132a3c4323834af78fecf29adbbbdd9d (DEFAULT_TABLE table=default.Table_1ef2001aff1f47f28501fb568a189e30 [id=e7fc3ccf3e3c41e4ae1c9efe872f72a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.444466 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f191436f3b9244eaa12d211b6f125175" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:27.444993 851 leader_election.cc:343] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.445237 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f191436f3b9244eaa12d211b6f125175" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.445713 850 leader_election.cc:343] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.445696 877 tablet_service.cc:1468] Processing CreateTablet for tablet 8d610d1c6d8f4f269387e56f099a782c (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.445840 850 leader_election.cc:304] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.446051 1793 raft_consensus.cc:2747] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:27.446749 985 leader_election.cc:343] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:27.446857 985 leader_election.cc:304] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:27.447254 1733 raft_consensus.cc:2747] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.447383 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d610d1c6d8f4f269387e56f099a782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.447553 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 132a3c4323834af78fecf29adbbbdd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.451145 1163 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.451272 1163 ts_tablet_manager.cc:1397] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250625 07:52:27.452000 1163 raft_consensus.cc:357] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.452163 1163 raft_consensus.cc:383] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.452212 1163 raft_consensus.cc:738] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.453150 1856 raft_consensus.cc:491] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.453303 1856 raft_consensus.cc:513] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.453621 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 581ac5caf07f4280801d5ec8b8c90b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.454002 1856 leader_election.cc:290] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.454319 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b056492dd84dbf9b4e7635e6751fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.455159 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 827) took 1470 ms. Trace:
I20250625 07:52:27.455248 1160 rpcz_store.cc:276] 0625 07:52:25.984633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:25.984680 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:27.344682 (+1360002us) ts_tablet_manager.cc:1145] Deleting tablet 77a127ebc7a74274b60cdd7d08ce22cb
0625 07:52:27.455032 (+110350us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.455139 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334695}
W20250625 07:52:27.455652 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 77a127ebc7a74274b60cdd7d08ce22cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77a127ebc7a74274b60cdd7d08ce22cb already in progress: creating tablet
I20250625 07:52:27.456032 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 836) took 1454 ms. Trace:
I20250625 07:52:27.456103 1160 rpcz_store.cc:276] 0625 07:52:26.001970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.002024 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:27.455927 (+1453903us) ts_tablet_manager.cc:1145] Deleting tablet 4a30941e3b9c475db17f961c38d96ef4
0625 07:52:27.455937 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.456017 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439809}
W20250625 07:52:27.456568 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4a30941e3b9c475db17f961c38d96ef4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a30941e3b9c475db17f961c38d96ef4 already in progress: creating tablet
I20250625 07:52:27.456967 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 837) took 1454 ms. Trace:
I20250625 07:52:27.457038 1160 rpcz_store.cc:276] 0625 07:52:26.002739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.002817 (+ 78us) service_pool.cc:225] Handling call
0625 07:52:27.456830 (+1454013us) ts_tablet_manager.cc:1145] Deleting tablet a6c0834adf1748c6a67805d26b973171
0625 07:52:27.456839 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.456950 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1440462}
W20250625 07:52:27.457399 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a6c0834adf1748c6a67805d26b973171 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c0834adf1748c6a67805d26b973171 already in progress: creating tablet
I20250625 07:52:27.457659 1160 tablet_replica.cc:331] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.457832 1160 raft_consensus.cc:2241] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.457976 1160 raft_consensus.cc:2270] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.458647 1160 ts_tablet_manager.cc:1905] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.459141 1793 raft_consensus.cc:491] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.459308 1793 raft_consensus.cc:513] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.460078 1793 leader_election.cc:290] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.460419 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e4bf78218f4b82b4f16f59b1704256" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.460755 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e4bf78218f4b82b4f16f59b1704256" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.460950 851 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:27.461306 850 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.461344 1163 consensus_queue.cc:260] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.461802 1163 ts_tablet_manager.cc:1428] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 07:52:27.461454 850 leader_election.cc:304] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.462054 1793 raft_consensus.cc:2747] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.463982 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d76681f162074b67a6b886887b5a3d61" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.464048 713 leader_election.cc:343] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d76681f162074b67a6b886887b5a3d61
W20250625 07:52:27.465728 713 leader_election.cc:343] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.465832 713 leader_election.cc:304] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.465911 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 960ded22e2eb42d9a3494c020a4ea354 (table default.Table_786d6b63b6904ffe87ce1f3eb9769954 [id=e8fb18d5b2ae454aa4f4a65fab67c467]) successfully deleted
I20250625 07:52:27.466107 1856 raft_consensus.cc:2747] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.467128 1158 tablet_bootstrap.cc:492] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.467255 1158 ts_tablet_manager.cc:1397] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.136s user 0.002s sys 0.000s
I20250625 07:52:27.467970 1158 raft_consensus.cc:357] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.468465 1158 raft_consensus.cc:383] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.468535 1158 raft_consensus.cc:738] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.468748 1158 consensus_queue.cc:260] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.469194 1158 ts_tablet_manager.cc:1428] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:27.470381 1158 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.462074 1163 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.482362 589 catalog_manager.cc:5582] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:27.482798 589 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:27.483047 589 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 4 got 0
I20250625 07:52:27.478545 1162 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.483639 1162 ts_tablet_manager.cc:1397] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.180s user 0.018s sys 0.018s
I20250625 07:52:27.483862 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5de0c5bf60b24808af124c95833d9c25
I20250625 07:52:27.484305 596 catalog_manager.cc:6652] ReplaceTablet: tablet 5de0c5bf60b24808af124c95833d9c25 deleted and replaced by tablet 67ea9c5fac904443a0e6004f8bebedbd
I20250625 07:52:27.484417 1162 raft_consensus.cc:357] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.484959 1162 raft_consensus.cc:383] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.485038 1162 raft_consensus.cc:738] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.485502 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de0c5bf60b24808af124c95833d9c25 with delete_type TABLET_DATA_DELETED (replaced by tablet 67ea9c5fac904443a0e6004f8bebedbd) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.468626 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8d610d1c6d8f4f269387e56f099a782c (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.485282 1162 consensus_queue.cc:260] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.485852 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d610d1c6d8f4f269387e56f099a782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.486066 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet c7a2b94af59648239cfd1e5ce5e8b968 with delete_type TABLET_DATA_DELETED (replaced by tablet 1228018baf3340d99b3f94b150d0d533) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.488538 596 master_service.cc:948] ReplaceTablet: received request to replace tablet edda0c3f60f64ed0b965eca52c5b3591 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:27.489833 1162 ts_tablet_manager.cc:1428] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:27.491323 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de0c5bf60b24808af124c95833d9c25 with delete_type TABLET_DATA_DELETED (replaced by tablet 67ea9c5fac904443a0e6004f8bebedbd) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.491480 1733 raft_consensus.cc:491] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.491626 1733 raft_consensus.cc:513] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.492213 1733 leader_election.cc:290] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:27.492550 1163 tablet_bootstrap.cc:654] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:27.493638 986 leader_election.cc:343] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b6382963863548398bed90f03b840806
W20250625 07:52:27.495095 985 leader_election.cc:343] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b6382963863548398bed90f03b840806
I20250625 07:52:27.495198 985 leader_election.cc:304] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:27.495920 1733 raft_consensus.cc:2747] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:27.496865 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eba09cd15d9f4611ab14d4a6dc372719 consensus state has no leader
I20250625 07:52:27.497829 1162 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.499511 1162 tablet_bootstrap.cc:654] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.500531 879 tablet_service.cc:1468] Processing CreateTablet for tablet 308b387d366c4d7884898937a9fc7690 (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.501024 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b387d366c4d7884898937a9fc7690. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.502748 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6009f781491c4481b5a2acdfdb2b8548 consensus state has no leader
I20250625 07:52:27.505034 755 tablet_service.cc:1468] Processing CreateTablet for tablet 308b387d366c4d7884898937a9fc7690 (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.505409 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b387d366c4d7884898937a9fc7690. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.504307 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5de0c5bf60b24808af124c95833d9c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de0c5bf60b24808af124c95833d9c25 already in progress: creating tablet
I20250625 07:52:27.505620 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_824016809113454d9d932a22a8d60512" } modify_external_catalogs: true
I20250625 07:52:27.505805 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_824016809113454d9d932a22a8d60512" } modify_external_catalogs: true
I20250625 07:52:27.510123 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_397ddcb2840845c48bc37edf6de92bbf" } modify_external_catalogs: true
I20250625 07:52:27.510267 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ac08bcc7329545ba8ba93e26d1c810f2" } modify_external_catalogs: true
I20250625 07:52:27.510335 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_397ddcb2840845c48bc37edf6de92bbf" } modify_external_catalogs: true
I20250625 07:52:27.511505 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ac08bcc7329545ba8ba93e26d1c810f2" } modify_external_catalogs: true
W20250625 07:52:27.513420 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ef2eb85358b414c9755f81f97beaddb consensus state has no leader
I20250625 07:52:27.513671 591 master_service.cc:948] ReplaceTablet: received request to replace tablet f9a94dc1432d4cbdbf3eecacb1429c23 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:27.514138 1160 ts_tablet_manager.cc:1918] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:27.515264 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2ee3980ed9747af87329fd3a1dfd699 has no consensus state
I20250625 07:52:27.515740 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 816) took 1094 ms. Trace:
I20250625 07:52:27.515830 1161 rpcz_store.cc:276] 0625 07:52:26.421628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.421675 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:27.515594 (+1093919us) ts_tablet_manager.cc:1145] Deleting tablet 7e46f27344b848318dcd1e782819b75e
0625 07:52:27.515607 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.515716 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1093771}
I20250625 07:52:27.516278 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8441cf3142424cc78e9fd8690dd8752e" } modify_external_catalogs: true
I20250625 07:52:27.516335 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 817) took 1077 ms. Trace:
I20250625 07:52:27.516398 1161 rpcz_store.cc:276] 0625 07:52:26.438745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.438807 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:27.516225 (+1077418us) ts_tablet_manager.cc:1145] Deleting tablet f73561bac3bf4ece9da5433ab4d19b4e
0625 07:52:27.516234 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.516318 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077203}
I20250625 07:52:27.516860 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 822) took 1005 ms. Trace:
I20250625 07:52:27.516868 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8441cf3142424cc78e9fd8690dd8752e" } modify_external_catalogs: true
I20250625 07:52:27.516932 1161 rpcz_store.cc:276] 0625 07:52:26.511805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.511856 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:27.516751 (+1004895us) ts_tablet_manager.cc:1145] Deleting tablet f5542f74374e4efe9acc6adc2c6e5b46
0625 07:52:27.516759 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.516841 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004731}
I20250625 07:52:27.517355 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 821) took 1005 ms. Trace:
I20250625 07:52:27.517422 1161 rpcz_store.cc:276] 0625 07:52:26.511652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.512080 (+ 428us) service_pool.cc:225] Handling call
0625 07:52:27.517262 (+1005182us) ts_tablet_manager.cc:1145] Deleting tablet daf115dd3aa548dc802ed0362c274d53
0625 07:52:27.517273 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.517341 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004956}
I20250625 07:52:27.517797 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 823) took 1005 ms. Trace:
I20250625 07:52:27.517864 1161 rpcz_store.cc:276] 0625 07:52:26.512806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.513621 (+ 815us) service_pool.cc:225] Handling call
0625 07:52:27.517706 (+1004085us) ts_tablet_manager.cc:1145] Deleting tablet 22411725aa5741c497328ce047c44d50
0625 07:52:27.517713 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.517781 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003777}
I20250625 07:52:27.518222 1161 tablet_replica.cc:331] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.518610 1161 raft_consensus.cc:2241] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.518765 1161 raft_consensus.cc:2270] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.519344 1161 ts_tablet_manager.cc:1905] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:27.519589 583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250625 07:52:27.519745 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca8eeef7078a4df39792b710bafcbd22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.520341 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca8eeef7078a4df39792b710bafcbd22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.520898 746 tablet_service.cc:1468] Processing CreateTablet for tablet cd49d80ed5b04d2abd0ead5231332a38 (DEFAULT_TABLE table=default.Table_b4005dc5ede948c992182a74c6f1e2c4 [id=dce7bdcd5aed45eb85f575f732e126ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.521389 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd49d80ed5b04d2abd0ead5231332a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.524474 743 tablet_service.cc:1468] Processing CreateTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.524849 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1a1685d87e45e1857af0b3532e7b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.524907 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7e4e88633c42a7b1bf70a4c64cea67 (DEFAULT_TABLE table=default.Table_c250beeb533e41af826edcc10557bf9a [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.525230 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7e4e88633c42a7b1bf70a4c64cea67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.528544 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3449de0badda4189b81bdad909bd368d (DEFAULT_TABLE table=default.Table_c7cdd85509b14915a9668f890f2ed721 [id=565111c2be19419da87f1a6c45920584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.528903 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3449de0badda4189b81bdad909bd368d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.533975 1158 tablet_bootstrap.cc:654] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.546888 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 231ced6803944e24829caf7853da017d
I20250625 07:52:27.547679 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet edda0c3f60f64ed0b965eca52c5b3591
I20250625 07:52:27.547858 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a396bde4b5964697ad0090eab777ba0e
I20250625 07:52:27.548149 596 catalog_manager.cc:6652] ReplaceTablet: tablet edda0c3f60f64ed0b965eca52c5b3591 deleted and replaced by tablet fab2dff56bc7445abcd6b7c1cb525553
I20250625 07:52:27.549244 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.549484 1166 tablet_replica.cc:331] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.549525 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a396bde4b5964697ad0090eab777ba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.549578 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 2f5fec0e5f364ca093879a8ad2e0bcfc
I20250625 07:52:27.549695 1166 raft_consensus.cc:2241] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.549885 1166 raft_consensus.cc:2270] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.550670 1166 ts_tablet_manager.cc:1905] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.553102 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.553161 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a630f8804bbf41de86b08cde419643ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.553735 743 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:27.555335 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet fcc3afc210a540058294121d2b83c66d (table default.Table_bb831997b8cf4b99a7779a4a6b23a238 [id=8a01186fa596413fb75529d0d57a8d9d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet fcc3afc210a540058294121d2b83c66d consensus state has no leader
I20250625 07:52:27.555639 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c28a064006145eea000e74da10d0794 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.516377 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.555931 1160 log.cc:1199] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/541c983d64f348539979f74199f79e59
I20250625 07:52:27.556092 749 tablet_service.cc:1515] Processing DeleteTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 with delete_type TABLET_DATA_DELETED (replaced by tablet fab2dff56bc7445abcd6b7c1cb525553) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.556279 1160 ts_tablet_manager.cc:1939] T 541c983d64f348539979f74199f79e59 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.556305 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 231ced6803944e24829caf7853da017d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.557749 1166 ts_tablet_manager.cc:1918] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.557895 1166 log.cc:1199] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:27.558248 1166 ts_tablet_manager.cc:1939] T 494cacf2dcbb40d3b2c9f05f64809870 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.558699 1158 tablet_bootstrap.cc:492] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.558845 1158 ts_tablet_manager.cc:1397] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20250625 07:52:27.558868 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f5a7f47ce2614d13bf092428963b3696
I20250625 07:52:27.559381 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 494cacf2dcbb40d3b2c9f05f64809870 (table default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]) successfully deleted
I20250625 07:52:27.559672 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 308b387d366c4d7884898937a9fc7690 (DEFAULT_TABLE table=default.Table_4aac2b881bfb4ada8db2169e25c54197 [id=d18d9df83d544a699bb3c675b92da24c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.559712 1158 raft_consensus.cc:357] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.559876 1158 raft_consensus.cc:383] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.559933 1158 raft_consensus.cc:738] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.560158 1158 consensus_queue.cc:260] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.560400 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 8d610d1c6d8f4f269387e56f099a782c (DEFAULT_TABLE table=default.Table_2c38b15750484886ad6bf5cc4253b510 [id=1df0b6c5d34546d1a06bf4251fc87e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.562839 1856 raft_consensus.cc:491] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.562964 1856 raft_consensus.cc:513] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.563364 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308b387d366c4d7884898937a9fc7690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.563500 1856 leader_election.cc:290] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.565425 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d610d1c6d8f4f269387e56f099a782c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.565865 1158 ts_tablet_manager.cc:1428] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:27.566146 1158 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.567664 1158 tablet_bootstrap.cc:654] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.570478 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.570598 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:27.570762 912 raft_consensus.cc:2466] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:27.571408 713 leader_election.cc:343] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.571468 1166 tablet_replica.cc:331] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.571635 1166 raft_consensus.cc:2241] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:27.571689 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a396bde4b5964697ad0090eab777ba0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a396bde4b5964697ad0090eab777ba0e already in progress: creating tablet
I20250625 07:52:27.571905 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 838) took 1564 ms. Trace:
W20250625 07:52:27.572589 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5fec0e5f364ca093879a8ad2e0bcfc already in progress: creating tablet
I20250625 07:52:27.573870 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7e4e88633c42a7b1bf70a4c64cea67 (DEFAULT_TABLE table=default.Table_c250beeb533e41af826edcc10557bf9a [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.574018 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3449de0badda4189b81bdad909bd368d (DEFAULT_TABLE table=default.Table_c7cdd85509b14915a9668f890f2ed721 [id=565111c2be19419da87f1a6c45920584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.574226 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7e4e88633c42a7b1bf70a4c64cea67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.574436 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3449de0badda4189b81bdad909bd368d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.572628 895 tablet_service.cc:1468] Processing CreateTablet for tablet cd49d80ed5b04d2abd0ead5231332a38 (DEFAULT_TABLE table=default.Table_b4005dc5ede948c992182a74c6f1e2c4 [id=dce7bdcd5aed45eb85f575f732e126ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.574486 1160 rpcz_store.cc:276] 0625 07:52:26.007596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.007649 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:27.457613 (+1449964us) ts_tablet_manager.cc:1145] Deleting tablet 541c983d64f348539979f74199f79e59
0625 07:52:27.457622 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.510924 (+ 53302us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.514040 (+ 3116us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.571888 (+ 57848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434970}
I20250625 07:52:27.574731 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd49d80ed5b04d2abd0ead5231332a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.574950 1160 tablet_replica.cc:331] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.575124 1160 raft_consensus.cc:2241] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.575258 1160 raft_consensus.cc:2270] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.575727 1160 ts_tablet_manager.cc:1905] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.577143 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_09bebf0eae58464c970e326392341524"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.577670 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09bebf0eae58464c970e326392341524 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.582281 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.582434 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 231ced6803944e24829caf7853da017d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.582603 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1a1685d87e45e1857af0b3532e7b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.583339 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 with delete_type TABLET_DATA_DELETED (replaced by tablet fab2dff56bc7445abcd6b7c1cb525553) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.583431 879 tablet_service.cc:1468] Processing CreateTablet for tablet 3449de0badda4189b81bdad909bd368d (DEFAULT_TABLE table=default.Table_c7cdd85509b14915a9668f890f2ed721 [id=565111c2be19419da87f1a6c45920584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.583736 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3449de0badda4189b81bdad909bd368d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.583755 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a396bde4b5964697ad0090eab777ba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.584159 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a396bde4b5964697ad0090eab777ba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.584390 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.584810 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.584834 892 tablet_service.cc:1515] Processing DeleteTablet for tablet edda0c3f60f64ed0b965eca52c5b3591 with delete_type TABLET_DATA_DELETED (replaced by tablet fab2dff56bc7445abcd6b7c1cb525553) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.585009 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 231ced6803944e24829caf7853da017d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.585681 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 541c983d64f348539979f74199f79e59 (table default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]) successfully deleted
W20250625 07:52:27.586424 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7e46f27344b848318dcd1e782819b75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e46f27344b848318dcd1e782819b75e already in progress: creating tablet
I20250625 07:52:27.573549 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8f7e4e88633c42a7b1bf70a4c64cea67 (DEFAULT_TABLE table=default.Table_c250beeb533e41af826edcc10557bf9a [id=86f69fa9857d43729a0f3ad1fc43b61d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:27.586763 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f73561bac3bf4ece9da5433ab4d19b4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73561bac3bf4ece9da5433ab4d19b4e already in progress: creating tablet
I20250625 07:52:27.586926 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7e4e88633c42a7b1bf70a4c64cea67. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:27.587004 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f5542f74374e4efe9acc6adc2c6e5b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5542f74374e4efe9acc6adc2c6e5b46 already in progress: creating tablet
W20250625 07:52:27.587193 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet daf115dd3aa548dc802ed0362c274d53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf115dd3aa548dc802ed0362c274d53 already in progress: creating tablet
W20250625 07:52:27.587426 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 22411725aa5741c497328ce047c44d50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22411725aa5741c497328ce047c44d50 already in progress: creating tablet
I20250625 07:52:27.593580 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_eab5eda71c3145cda2327c5ffb28fb81" } new_table_name: "default.Table_f260806f3d3d4af59e7c35222a769938" modify_external_catalogs: true
I20250625 07:52:27.594882 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_602ca03b4f824a7686588dc776b8bed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.595433 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_602ca03b4f824a7686588dc776b8bed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.598237 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 29eaa8d93e5b4d1d8ed147b60c7da859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:27.598901 583 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 00a3faeffaa4422ea73b98ee74768a4a (127.0.104.62:38477): Couldn't send request to peer 00a3faeffaa4422ea73b98ee74768a4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38477: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250625 07:52:27.599792 1163 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.599929 1163 ts_tablet_manager.cc:1397] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.000s
I20250625 07:52:27.600488 1161 ts_tablet_manager.cc:1918] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.601186 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_98f0756428904e40a7406141b5194756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.601763 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f0756428904e40a7406141b5194756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.572940 713 leader_election.cc:304] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.598780 1162 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.602385 1850 raft_consensus.cc:2802] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:27.602448 1162 ts_tablet_manager.cc:1397] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.105s user 0.004s sys 0.001s
I20250625 07:52:27.602464 1850 raft_consensus.cc:491] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:27.602556 1850 raft_consensus.cc:3058] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.602937 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2f8acc41371a4800a82de306adce74db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.603219 1162 raft_consensus.cc:357] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.603366 1162 raft_consensus.cc:383] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.603410 1162 raft_consensus.cc:738] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:27.603405 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f8acc41371a4800a82de306adce74db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.603581 1162 consensus_queue.cc:260] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.603997 1162 ts_tablet_manager.cc:1428] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:27.604291 1162 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.565258 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 5de0c5bf60b24808af124c95833d9c25 with delete_type TABLET_DATA_DELETED (replaced by tablet 67ea9c5fac904443a0e6004f8bebedbd) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.604796 1166 raft_consensus.cc:2270] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.605070 1850 raft_consensus.cc:513] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.606171 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:27.606340 912 raft_consensus.cc:3058] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.606940 1162 tablet_bootstrap.cc:654] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.574386 1029 tablet_service.cc:1468] Processing CreateTablet for tablet cd49d80ed5b04d2abd0ead5231332a38 (DEFAULT_TABLE table=default.Table_b4005dc5ede948c992182a74c6f1e2c4 [id=dce7bdcd5aed45eb85f575f732e126ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.608147 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b36659e6e06d41298f3a614061f97b73" } modify_external_catalogs: true
I20250625 07:52:27.608299 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b36659e6e06d41298f3a614061f97b73" } modify_external_catalogs: true
I20250625 07:52:27.608404 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1228018baf3340d99b3f94b150d0d533" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:27.605727 1850 leader_election.cc:290] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.609267 912 raft_consensus.cc:2466] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:27.609688 1163 raft_consensus.cc:357] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.609862 1163 raft_consensus.cc:383] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.609836 713 leader_election.cc:304] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:27.609910 1163 raft_consensus.cc:738] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:27.610087 713 leader_election.cc:343] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.610100 1163 consensus_queue.cc:260] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.610379 1850 raft_consensus.cc:2802] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:27.610735 1850 raft_consensus.cc:695] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:27.610965 1850 consensus_queue.cc:237] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:27.614392 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 059dad944716497bbba17e11fb3ba5be has no consensus state
I20250625 07:52:27.618464 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.619493 1160 ts_tablet_manager.cc:1918] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.619541 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd49d80ed5b04d2abd0ead5231332a38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.622455 1166 ts_tablet_manager.cc:1905] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.628994 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.629386 1162 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.629483 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c1a1685d87e45e1857af0b3532e7b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.633771 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:27.636204 1158 tablet_bootstrap.cc:492] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.637025 1158 ts_tablet_manager.cc:1397] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.002s
I20250625 07:52:27.632261 1161 log.cc:1199] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9c770b41e0844786ad81c7b3579f9934
I20250625 07:52:27.638043 1161 ts_tablet_manager.cc:1939] T 9c770b41e0844786ad81c7b3579f9934 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.638368 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 824) took 1115 ms. Trace:
I20250625 07:52:27.638450 1161 rpcz_store.cc:276] 0625 07:52:26.522417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.522467 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:27.518156 (+995689us) ts_tablet_manager.cc:1145] Deleting tablet 9c770b41e0844786ad81c7b3579f9934
0625 07:52:27.518167 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.565369 (+ 47202us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.600396 (+ 35027us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.638354 (+ 37958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995444}
I20250625 07:52:27.638958 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9c770b41e0844786ad81c7b3579f9934 (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:27.629729 1166 ts_tablet_manager.cc:1918] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.639405 1166 log.cc:1199] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a630f8804bbf41de86b08cde419643ff
I20250625 07:52:27.639408 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 827) took 1004 ms. Trace:
I20250625 07:52:27.639488 1161 rpcz_store.cc:276] 0625 07:52:26.634732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.635451 (+ 719us) service_pool.cc:225] Handling call
0625 07:52:27.639264 (+1003813us) ts_tablet_manager.cc:1145] Deleting tablet 8f09c9fd1b3a4dae9c8b366000737b5e
0625 07:52:27.639278 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.639390 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003562}
I20250625 07:52:27.638996 1158 raft_consensus.cc:357] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.639724 1166 ts_tablet_manager.cc:1939] T a630f8804bbf41de86b08cde419643ff P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.639904 1158 raft_consensus.cc:383] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:27.639910 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8f09c9fd1b3a4dae9c8b366000737b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f09c9fd1b3a4dae9c8b366000737b5e already in progress: creating tablet
I20250625 07:52:27.640127 1158 raft_consensus.cc:738] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.640573 1166 tablet_replica.cc:331] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.640787 1166 raft_consensus.cc:2241] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.640532 1158 consensus_queue.cc:260] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.640938 1166 raft_consensus.cc:2270] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.640202 1161 tablet_replica.cc:331] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.641316 1161 raft_consensus.cc:2241] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.641443 1161 raft_consensus.cc:2270] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.641883 1161 ts_tablet_manager.cc:1905] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.642532 1160 log.cc:1199] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:27.642745 1166 ts_tablet_manager.cc:1905] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.642869 1160 ts_tablet_manager.cc:1939] T f9a94dc1432d4cbdbf3eecacb1429c23 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.643404 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a630f8804bbf41de86b08cde419643ff (table default.Table_929703cfa29e467c9090ff84a8b2cec1 [id=cd9d83bd28e5422188207277162018a1]) successfully deleted
I20250625 07:52:27.644353 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 840) took 1387 ms. Trace:
I20250625 07:52:27.644608 1158 ts_tablet_manager.cc:1428] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:27.646204 1158 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.646297 1160 rpcz_store.cc:276] 0625 07:52:26.256967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.257023 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:27.574907 (+1317884us) ts_tablet_manager.cc:1145] Deleting tablet f9a94dc1432d4cbdbf3eecacb1429c23
0625 07:52:27.574918 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.616566 (+ 41648us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.619405 (+ 2839us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.644331 (+ 24926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317685}
I20250625 07:52:27.647313 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4f1c150fb54c405c8dc61ac133d98b13
I20250625 07:52:27.647964 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.629873 1793 raft_consensus.cc:491] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.648833 1162 ts_tablet_manager.cc:1397] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20250625 07:52:27.648774 1793 raft_consensus.cc:513] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.649560 1793 leader_election.cc:290] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.649744 1162 raft_consensus.cc:357] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.649921 1162 raft_consensus.cc:383] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.649987 1162 raft_consensus.cc:738] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.650018 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet f9a94dc1432d4cbdbf3eecacb1429c23 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
I20250625 07:52:27.650398 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 841) took 1385 ms. Trace:
I20250625 07:52:27.650437 1166 ts_tablet_manager.cc:1918] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.650537 1166 log.cc:1199] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/d76681f162074b67a6b886887b5a3d61
I20250625 07:52:27.650462 1160 rpcz_store.cc:276] 0625 07:52:26.264662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.264736 (+ 74us) service_pool.cc:225] Handling call
0625 07:52:27.650275 (+1385539us) ts_tablet_manager.cc:1145] Deleting tablet 1764c3d48a7546f7828ee93aa51dcb51
0625 07:52:27.650285 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.650383 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385348}
I20250625 07:52:27.650848 1166 ts_tablet_manager.cc:1939] T d76681f162074b67a6b886887b5a3d61 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:27.650838 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1764c3d48a7546f7828ee93aa51dcb51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1764c3d48a7546f7828ee93aa51dcb51 already in progress: creating tablet
I20250625 07:52:27.629607 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.648557 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.651239 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 842) took 1377 ms. Trace:
I20250625 07:52:27.651301 1160 rpcz_store.cc:276] 0625 07:52:26.274194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.274251 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:27.651134 (+1376883us) ts_tablet_manager.cc:1145] Deleting tablet b6661b294489423ab06fa7c6988b75f7
0625 07:52:27.651142 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.651217 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376711}
W20250625 07:52:27.651683 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b6661b294489423ab06fa7c6988b75f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6661b294489423ab06fa7c6988b75f7 already in progress: creating tablet
I20250625 07:52:27.651803 1166 tablet_replica.cc:331] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.651996 1166 raft_consensus.cc:2241] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:27.652114 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 844) took 1376 ms. Trace:
I20250625 07:52:27.652175 1160 rpcz_store.cc:276] 0625 07:52:26.275657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.275703 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:27.651941 (+1376238us) ts_tablet_manager.cc:1145] Deleting tablet d1fc7cb613b6472ea34a0ea7bc8a08ea
0625 07:52:27.651952 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.652100 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376072}
I20250625 07:52:27.652246 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7480c07af7ce4a3889bd8d42d7419d57" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.652316 1166 raft_consensus.cc:2270] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:27.652510 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d1fc7cb613b6472ea34a0ea7bc8a08ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fc7cb613b6472ea34a0ea7bc8a08ea already in progress: creating tablet
I20250625 07:52:27.652791 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7480c07af7ce4a3889bd8d42d7419d57" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.653008 1162 consensus_queue.cc:260] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.653481 1162 ts_tablet_manager.cc:1428] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20250625 07:52:27.655892 850 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:27.656096 851 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.656198 851 leader_election.cc:304] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.656415 1793 raft_consensus.cc:2747] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.656633 1160 tablet_replica.cc:331] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.653724 1162 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.657827 1160 raft_consensus.cc:2241] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.660295 1162 tablet_bootstrap.cc:654] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.661082 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet d76681f162074b67a6b886887b5a3d61 (table default.Table_96d96b9fee4d48efaeb2f22ff67b43e6 [id=8340c9a7e5874bf486d4f07192ebb908]) successfully deleted
I20250625 07:52:27.663682 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a777f64425384be5bccc86b197626618 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e5a207d515449e298936d8bb532437a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.665776 1158 tablet_bootstrap.cc:654] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.657979 1160 raft_consensus.cc:2270] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.667347 1160 ts_tablet_manager.cc:1905] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.633651 1163 ts_tablet_manager.cc:1428] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20250625 07:52:27.673867 1163 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.662596 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.671360 1166 ts_tablet_manager.cc:1905] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.679615 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_006e977b28ff4dbe9f2091ba6f1879e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.681190 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_006e977b28ff4dbe9f2091ba6f1879e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.681826 1162 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:27.682345 1162 ts_tablet_manager.cc:1397] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20250625 07:52:27.683359 1925 raft_consensus.cc:491] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.683496 1925 raft_consensus.cc:513] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.683565 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.684072 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.684118 1925 leader_election.cc:290] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.685752 1158 tablet_bootstrap.cc:492] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.685880 1158 ts_tablet_manager.cc:1397] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.005s
I20250625 07:52:27.686610 1158 raft_consensus.cc:357] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.686765 1158 raft_consensus.cc:383] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.686810 1158 raft_consensus.cc:738] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.686802 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b9491eb269948a29888a7301ea75d42" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.687081 1158 consensus_queue.cc:260] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:27.687312 850 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.690627 1158 ts_tablet_manager.cc:1428] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:27.693482 1158 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.695149 1158 tablet_bootstrap.cc:654] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.696595 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet b377c8111c9f4ac2ba05e65694d611b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 89c703dc3249408faeb522cef0248111) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.698376 1163 tablet_bootstrap.cc:654] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.699044 1161 ts_tablet_manager.cc:1918] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.699151 1161 log.cc:1199] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b6382963863548398bed90f03b840806
I20250625 07:52:27.699445 1161 ts_tablet_manager.cc:1939] T b6382963863548398bed90f03b840806 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.699715 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 828) took 1036 ms. Trace:
I20250625 07:52:27.699790 1161 rpcz_store.cc:276] 0625 07:52:26.663047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.664653 (+ 1606us) service_pool.cc:225] Handling call
0625 07:52:27.640158 (+975505us) ts_tablet_manager.cc:1145] Deleting tablet b6382963863548398bed90f03b840806
0625 07:52:27.640168 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.661063 (+ 20895us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.698925 (+ 37862us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.699702 (+ 777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":975202}
I20250625 07:52:27.700098 1161 tablet_replica.cc:331] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.700254 1161 raft_consensus.cc:2241] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.700390 1161 raft_consensus.cc:2270] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.700865 1161 ts_tablet_manager.cc:1905] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.701323 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b6382963863548398bed90f03b840806 (table default.Table_7c4e84ad77a74e08a1c7aab848d552b7 [id=091524e3dfd643aab134494c04a26880]) successfully deleted
I20250625 07:52:27.704597 1158 tablet_bootstrap.cc:492] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.704770 1158 ts_tablet_manager.cc:1397] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20250625 07:52:27.705466 1158 raft_consensus.cc:357] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.705665 1158 raft_consensus.cc:383] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.705834 1158 raft_consensus.cc:738] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.706090 1158 consensus_queue.cc:260] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.707187 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f001237d203b42898ff32eff812b5557" } new_table_name: "default.Table_da048cffd3284b26bcbbba35f2a14300" modify_external_catalogs: true
I20250625 07:52:27.708518 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1c27389c187843de9945c7a9e3d1bed4" } new_table_name: "default.Table_5da3464e644f46ed9a0bd81795853dec" modify_external_catalogs: true
I20250625 07:52:27.711602 1158 ts_tablet_manager.cc:1428] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:27.713524 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b2e47db078dd4a8c8ec4c8df33955362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.713886 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d9d66546d7c42949dfe5db77f1057fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.686416 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b9491eb269948a29888a7301ea75d42" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:27.714962 1162 raft_consensus.cc:357] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.715116 1162 raft_consensus.cc:383] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.715171 1162 raft_consensus.cc:738] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:27.715400 851 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.715366 1162 consensus_queue.cc:260] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.715508 851 leader_election.cc:304] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.715746 1925 raft_consensus.cc:2747] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.715791 1162 ts_tablet_manager.cc:1428] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:27.715930 1158 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.717520 1158 tablet_bootstrap.cc:654] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.717810 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 32a9f4cf6838411fab06fa6d35ae9492 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:27.718046 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d3a5e89e18914595b2f605c893c2d050" } new_table_name: "default.Table_41ae91a0d4a246c9a9a414334e6e257c" modify_external_catalogs: true
I20250625 07:52:27.718317 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1a052ae2f0774e85bdc42172e61770d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.723040 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a052ae2f0774e85bdc42172e61770d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.726686 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_457927be3bfb45d1ab90fc5a3b8163d8" } new_table_name: "default.Table_303504fb62ac468c8985d6d1b3f25851" modify_external_catalogs: true
W20250625 07:52:27.719048 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8d9333ac16747b48830505bb5f1c2d5 consensus state has no leader
I20250625 07:52:27.720134 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_90f97f84512542b29acecb9db3e1172d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.728554 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90f97f84512542b29acecb9db3e1172d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.729264 1163 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.729404 1163 ts_tablet_manager.cc:1397] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.005s
I20250625 07:52:27.730257 1163 raft_consensus.cc:357] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.730427 1163 raft_consensus.cc:383] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.730481 1163 raft_consensus.cc:738] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.730661 1163 consensus_queue.cc:260] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.731231 1163 ts_tablet_manager.cc:1428] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:27.731493 1163 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.733088 1163 tablet_bootstrap.cc:654] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:27.738207 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 550e64a19120444a8e7b27db9bbf09f5 consensus state has no leader
I20250625 07:52:27.739392 1162 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:27.743557 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2e47db078dd4a8c8ec4c8df33955362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.744326 1166 ts_tablet_manager.cc:1918] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:27.749941 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04" } modify_external_catalogs: true
I20250625 07:52:27.750145 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04" } modify_external_catalogs: true
W20250625 07:52:27.751415 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8b5bb3fe3ef4bfdba93b862784a9f61 consensus state has no leader
I20250625 07:52:27.752157 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1f8502fb577748268f87ac63e15ec4f5" } modify_external_catalogs: true
I20250625 07:52:27.752350 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1f8502fb577748268f87ac63e15ec4f5" } modify_external_catalogs: true
I20250625 07:52:27.752761 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1fb8a396533b4c55ab45e2dbd46237d7" } modify_external_catalogs: true
I20250625 07:52:27.752956 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1fb8a396533b4c55ab45e2dbd46237d7" } modify_external_catalogs: true
I20250625 07:52:27.748833 1850 raft_consensus.cc:491] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.756121 1166 log.cc:1199] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/edda0c3f60f64ed0b965eca52c5b3591
I20250625 07:52:27.756153 1850 raft_consensus.cc:513] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.756490 1166 ts_tablet_manager.cc:1939] T edda0c3f60f64ed0b965eca52c5b3591 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.755280 1162 tablet_bootstrap.cc:654] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.757220 1850 leader_election.cc:290] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.758248 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:27.758345 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:27.758582 912 raft_consensus.cc:2466] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:27.758965 713 leader_election.cc:343] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.759383 713 leader_election.cc:304] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.759789 1850 raft_consensus.cc:2802] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:27.759862 1850 raft_consensus.cc:491] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:27.759927 1850 raft_consensus.cc:3058] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.760391 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_94af0d30f3b8498c8bd98dbeec6e5085" } modify_external_catalogs: true
I20250625 07:52:27.760610 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_94af0d30f3b8498c8bd98dbeec6e5085" } modify_external_catalogs: true
I20250625 07:52:27.762336 1733 raft_consensus.cc:491] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.762470 1733 raft_consensus.cc:513] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.762782 1850 raft_consensus.cc:513] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.763047 1733 leader_election.cc:290] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:27.763916 986 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7d9d66546d7c42949dfe5db77f1057fe
W20250625 07:52:27.764322 985 leader_election.cc:343] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:27.764428 985 leader_election.cc:304] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:27.764587 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:27.764770 912 raft_consensus.cc:3058] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:27.764827 1733 raft_consensus.cc:2747] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.765928 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb23f900faf6421fb6e72f2b5202d0fd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:27.766383 713 leader_election.cc:343] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:27.766520 1850 leader_election.cc:290] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.767257 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3b5acb6fb6cd40d2855892e2d28bed37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.768249 912 raft_consensus.cc:2466] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:27.768898 713 leader_election.cc:304] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.769580 1856 raft_consensus.cc:491] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:27.769704 1856 raft_consensus.cc:513] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.770244 1856 leader_election.cc:290] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:27.770380 1856 raft_consensus.cc:2802] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:27.770462 1856 raft_consensus.cc:695] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:27.770687 1856 consensus_queue.cc:237] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.772042 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]) successfully deleted
I20250625 07:52:27.772364 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6239713a586d43c28c101ba2447d579d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:27.773219 713 leader_election.cc:343] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:27.773394 713 leader_election.cc:343] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6239713a586d43c28c101ba2447d579d
I20250625 07:52:27.773466 713 leader_election.cc:304] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:27.776821 1166 tablet_replica.cc:331] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:27.777694 1166 raft_consensus.cc:2241] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.778401 1166 raft_consensus.cc:2270] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.778733 1850 raft_consensus.cc:2747] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:27.789590 1162 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:27.795843 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b5acb6fb6cd40d2855892e2d28bed37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.797605 1160 ts_tablet_manager.cc:1918] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.798296 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fa33440d4d5740c38f6730f962447c3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.798888 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa33440d4d5740c38f6730f962447c3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.804862 1162 ts_tablet_manager.cc:1397] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.004s
I20250625 07:52:27.795293 1166 ts_tablet_manager.cc:1905] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.812817 1158 tablet_bootstrap.cc:492] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.813066 1158 ts_tablet_manager.cc:1397] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.042s
I20250625 07:52:27.813949 1158 raft_consensus.cc:357] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.814186 1158 raft_consensus.cc:383] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.814293 1158 raft_consensus.cc:738] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.814553 1158 consensus_queue.cc:260] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.815431 1161 ts_tablet_manager.cc:1918] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.815529 1161 log.cc:1199] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/49697a10cdac4c8ebb69f10f4c3437ec
I20250625 07:52:27.815795 1161 ts_tablet_manager.cc:1939] T 49697a10cdac4c8ebb69f10f4c3437ec P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:27.816233 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 231ced6803944e24829caf7853da017d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 231ced6803944e24829caf7853da017d already in progress: creating tablet
I20250625 07:52:27.816826 1158 ts_tablet_manager.cc:1428] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:27.817874 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f678232d9ed444da9031bda1a4bf561e" } new_table_name: "default.Table_b8b158d4a1bd40e3ba6c1f74038a5291" modify_external_catalogs: true
I20250625 07:52:27.818655 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 81976db0a2f94128a89842a12f067231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.798508 1163 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.818941 1163 ts_tablet_manager.cc:1397] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.088s user 0.021s sys 0.008s
I20250625 07:52:27.819698 1163 raft_consensus.cc:357] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.819859 1163 raft_consensus.cc:383] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.819904 1163 raft_consensus.cc:738] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.820130 1163 consensus_queue.cc:260] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.820617 1163 ts_tablet_manager.cc:1428] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:27.821015 1163 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.805907 1162 raft_consensus.cc:357] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.821811 1162 raft_consensus.cc:383] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.821913 1162 raft_consensus.cc:738] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.822381 1162 consensus_queue.cc:260] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.822712 1163 tablet_bootstrap.cc:654] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.823168 1162 ts_tablet_manager.cc:1428] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250625 07:52:27.825067 1160 log.cc:1199] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a630f8804bbf41de86b08cde419643ff
I20250625 07:52:27.825563 1160 ts_tablet_manager.cc:1939] T a630f8804bbf41de86b08cde419643ff P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.826665 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 845) took 1550 ms. Trace:
W20250625 07:52:27.827064 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d705b102654747b5a9f16ae9290f96b4 consensus state has no leader
I20250625 07:52:27.830236 588 master_service.cc:948] ReplaceTablet: received request to replace tablet cbdbd9230210455ba615d51f1a98267b from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:27.832896 1160 rpcz_store.cc:276] 0625 07:52:26.276004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.276086 (+ 82us) service_pool.cc:225] Handling call
0625 07:52:27.656595 (+1380509us) ts_tablet_manager.cc:1145] Deleting tablet a630f8804bbf41de86b08cde419643ff
0625 07:52:27.656604 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.761738 (+105134us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.797443 (+ 35705us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.826646 (+ 29203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380338}
I20250625 07:52:27.833398 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 843) took 1557 ms. Trace:
I20250625 07:52:27.833494 1160 rpcz_store.cc:276] 0625 07:52:26.275554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.276298 (+ 744us) service_pool.cc:225] Handling call
0625 07:52:27.833248 (+1556950us) ts_tablet_manager.cc:1145] Deleting tablet 670f3b7f0a904656af418d01ed6f58b4
0625 07:52:27.833256 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.833377 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1556804}
I20250625 07:52:27.833775 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a630f8804bbf41de86b08cde419643ff (table default.Table_929703cfa29e467c9090ff84a8b2cec1 [id=cd9d83bd28e5422188207277162018a1]) successfully deleted
I20250625 07:52:27.833854 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 846) took 1539 ms. Trace:
I20250625 07:52:27.833943 1160 rpcz_store.cc:276] 0625 07:52:26.294498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.294554 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:27.833734 (+1539180us) ts_tablet_manager.cc:1145] Deleting tablet e73689f413504e28ba97a83ab7b03fa1
0625 07:52:27.833745 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.833841 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538986}
W20250625 07:52:27.834214 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 670f3b7f0a904656af418d01ed6f58b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670f3b7f0a904656af418d01ed6f58b4 already in progress: creating tablet
W20250625 07:52:27.834748 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e73689f413504e28ba97a83ab7b03fa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e73689f413504e28ba97a83ab7b03fa1 already in progress: creating tablet
I20250625 07:52:27.839079 1163 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.834322 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 848) took 1494 ms. Trace:
I20250625 07:52:27.840994 1163 ts_tablet_manager.cc:1397] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20250625 07:52:27.840905 1160 rpcz_store.cc:276] 0625 07:52:26.339950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.340046 (+ 96us) service_pool.cc:225] Handling call
0625 07:52:27.834185 (+1494139us) ts_tablet_manager.cc:1145] Deleting tablet 2398a407b3154334a6a167946cf99ad7
0625 07:52:27.834196 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.834307 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1493894}
W20250625 07:52:27.841496 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2398a407b3154334a6a167946cf99ad7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2398a407b3154334a6a167946cf99ad7 already in progress: creating tablet
I20250625 07:52:27.841883 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 850) took 1491 ms. Trace:
I20250625 07:52:27.842006 1160 rpcz_store.cc:276] 0625 07:52:26.350484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.350539 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:27.841774 (+1491235us) ts_tablet_manager.cc:1145] Deleting tablet a0342575a2b34bf4b2390968e6acbe5a
0625 07:52:27.841783 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.841866 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490990}
I20250625 07:52:27.842056 1163 raft_consensus.cc:357] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.842309 1163 raft_consensus.cc:383] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.842372 1163 raft_consensus.cc:738] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:27.842614 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a0342575a2b34bf4b2390968e6acbe5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0342575a2b34bf4b2390968e6acbe5a already in progress: creating tablet
I20250625 07:52:27.842588 1163 consensus_queue.cc:260] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:27.843179 1163 ts_tablet_manager.cc:1428] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:27.843539 1163 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:27.845600 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 852) took 1425 ms. Trace:
I20250625 07:52:27.845767 1163 tablet_bootstrap.cc:654] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.845734 1160 rpcz_store.cc:276] 0625 07:52:26.419989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.420044 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:27.845485 (+1425441us) ts_tablet_manager.cc:1145] Deleting tablet 7e46f27344b848318dcd1e782819b75e
0625 07:52:27.845494 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.845582 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425263}
W20250625 07:52:27.846376 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7e46f27344b848318dcd1e782819b75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e46f27344b848318dcd1e782819b75e already in progress: creating tablet
I20250625 07:52:27.846817 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 854) took 1407 ms. Trace:
I20250625 07:52:27.846951 1160 rpcz_store.cc:276] 0625 07:52:26.439202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.439251 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:27.846718 (+1407467us) ts_tablet_manager.cc:1145] Deleting tablet f73561bac3bf4ece9da5433ab4d19b4e
0625 07:52:27.846728 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.846802 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1407225}
W20250625 07:52:27.847452 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f73561bac3bf4ece9da5433ab4d19b4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f73561bac3bf4ece9da5433ab4d19b4e already in progress: creating tablet
I20250625 07:52:27.847822 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 859) took 1333 ms. Trace:
I20250625 07:52:27.847985 1160 rpcz_store.cc:276] 0625 07:52:26.514364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.514414 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:27.847730 (+1333316us) ts_tablet_manager.cc:1145] Deleting tablet daf115dd3aa548dc802ed0362c274d53
0625 07:52:27.847738 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.847807 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333136}
W20250625 07:52:27.848506 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet daf115dd3aa548dc802ed0362c274d53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf115dd3aa548dc802ed0362c274d53 already in progress: creating tablet
I20250625 07:52:27.848878 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 860) took 1333 ms. Trace:
I20250625 07:52:27.849468 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6aceb5dc89a54074b4b33990008cb85e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.849747 1160 rpcz_store.cc:276] 0625 07:52:26.515444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.515490 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:27.848790 (+1333300us) ts_tablet_manager.cc:1145] Deleting tablet 22411725aa5741c497328ce047c44d50
0625 07:52:27.848798 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.848864 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333146}
W20250625 07:52:27.850171 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aceb5dc89a54074b4b33990008cb85e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:27.850306 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 22411725aa5741c497328ce047c44d50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22411725aa5741c497328ce047c44d50 already in progress: creating tablet
I20250625 07:52:27.850682 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 858) took 1336 ms. Trace:
I20250625 07:52:27.850845 1160 rpcz_store.cc:276] 0625 07:52:26.514250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.515742 (+ 1492us) service_pool.cc:225] Handling call
0625 07:52:27.850577 (+1334835us) ts_tablet_manager.cc:1145] Deleting tablet f5542f74374e4efe9acc6adc2c6e5b46
0625 07:52:27.850586 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.850662 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334661}
W20250625 07:52:27.851485 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f5542f74374e4efe9acc6adc2c6e5b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5542f74374e4efe9acc6adc2c6e5b46 already in progress: creating tablet
I20250625 07:52:27.852058 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 829) took 1138 ms. Trace:
I20250625 07:52:27.852150 1161 rpcz_store.cc:276] 0625 07:52:26.713703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.715775 (+ 2072us) service_pool.cc:225] Handling call
0625 07:52:27.700059 (+984284us) ts_tablet_manager.cc:1145] Deleting tablet 49697a10cdac4c8ebb69f10f4c3437ec
0625 07:52:27.700067 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.703743 (+ 3676us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.815336 (+111593us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.852036 (+ 36700us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2304,"tablet-delete.queue_time_us":984071}
I20250625 07:52:27.852599 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 830) took 1126 ms. Trace:
I20250625 07:52:27.853188 1158 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.856215 1158 tablet_bootstrap.cc:654] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.857443 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 49697a10cdac4c8ebb69f10f4c3437ec (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:27.852665 1161 rpcz_store.cc:276] 0625 07:52:26.725832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.725883 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:27.852495 (+1126612us) ts_tablet_manager.cc:1145] Deleting tablet 8c2ced3d2fba4f74a5d89d4bc5580905
0625 07:52:27.852503 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.852588 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126438}
I20250625 07:52:27.858193 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 831) took 1105 ms. Trace:
I20250625 07:52:27.858281 1161 rpcz_store.cc:276] 0625 07:52:26.753233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.753379 (+ 146us) service_pool.cc:225] Handling call
0625 07:52:27.858053 (+1104674us) ts_tablet_manager.cc:1145] Deleting tablet d605113ad8f246148825923e5bcdf00d
0625 07:52:27.858067 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.858174 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104428}
W20250625 07:52:27.858958 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c2ced3d2fba4f74a5d89d4bc5580905 already in progress: creating tablet
W20250625 07:52:27.859269 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d605113ad8f246148825923e5bcdf00d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d605113ad8f246148825923e5bcdf00d already in progress: creating tablet
I20250625 07:52:27.859586 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 832) took 1081 ms. Trace:
I20250625 07:52:27.859704 1161 rpcz_store.cc:276] 0625 07:52:26.777854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.778089 (+ 235us) service_pool.cc:225] Handling call
0625 07:52:27.858776 (+1080687us) ts_tablet_manager.cc:1145] Deleting tablet c8c015c202544914b74f1e245b496f82
0625 07:52:27.859486 (+ 710us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.859569 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080472}
I20250625 07:52:27.859980 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e2e1925e33414e199b380b98a0c6e96f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:27.860118 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f62f60f959ba405b8f9f91731541cee3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.860601 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2e1925e33414e199b380b98a0c6e96f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:27.860661 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f62f60f959ba405b8f9f91731541cee3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:27.860800 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c8c015c202544914b74f1e245b496f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8c015c202544914b74f1e245b496f82 already in progress: creating tablet
W20250625 07:52:27.861063 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f04016cda7f746d49e30be3b1c9c18b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f04016cda7f746d49e30be3b1c9c18b6 already in progress: creating tablet
W20250625 07:52:27.861336 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4167282133f94bfb9870bd08fc6594b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4167282133f94bfb9870bd08fc6594b4 already in progress: creating tablet
W20250625 07:52:27.862213 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5b0d357cefb84a239d5be73857b9097b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0d357cefb84a239d5be73857b9097b already in progress: creating tablet
W20250625 07:52:27.862600 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 52c1f7cdd88d423788d7d30f15318786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c1f7cdd88d423788d7d30f15318786 already in progress: creating tablet
I20250625 07:52:27.862934 1160 tablet_replica.cc:331] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.863174 1160 raft_consensus.cc:2241] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.863396 1160 raft_consensus.cc:2270] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.862222 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c40382d89af6427eaf3c73a32b21d835" } modify_external_catalogs: true
I20250625 07:52:27.865007 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c40382d89af6427eaf3c73a32b21d835" } modify_external_catalogs: true
I20250625 07:52:27.865478 1161 tablet_replica.cc:331] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:27.865770 1161 raft_consensus.cc:2241] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.865921 1161 raft_consensus.cc:2270] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.866410 1161 ts_tablet_manager.cc:1905] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.875093 593 catalog_manager.cc:5582] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:27.875653 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 4 got 0
I20250625 07:52:27.877133 1945 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:27.877869 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
W20250625 07:52:27.890142 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ccd929ffd026433c9abd3851bb8e1004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccd929ffd026433c9abd3851bb8e1004 already in progress: creating tablet
I20250625 07:52:27.865140 1160 ts_tablet_manager.cc:1905] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:27.882419 1948 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:27.897471 1162 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:27.898062 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8319e385441549e98fc4f6f600dd17b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.898700 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8319e385441549e98fc4f6f600dd17b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.901019 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_895827fb99994402b75be290df98a2ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.901645 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_895827fb99994402b75be290df98a2ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.901875 1944 tablet.cc:1722] T 6b3f2915b351407da08a9d6ba8ed402f P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:27.903682 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6d1017546c484c9a89ec039b91c63ad4
I20250625 07:52:27.904795 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 32a2929bad2340db9e6e0b5364ec8567
I20250625 07:52:27.904771 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.905723 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:27.905764 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.906483 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.903076 1162 tablet_bootstrap.cc:654] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.911368 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.911554 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.911633 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.918460 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 81451d3745f24cbd9a56307c557e5ec1
I20250625 07:52:27.921010 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.928270 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.929447 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 9f4359fd842a4760bae75169e8914f1c (table default.Table_56a7bd0b475e451b8afa385ff9915ba5 [id=914805a4f6684943a2e76c948377f326]). Expected version 1 got 0
I20250625 07:52:27.929867 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1afe90251c964e369d05e82c7db19326 (table default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]). Expected version 1 got 0
I20250625 07:52:27.930387 1166 ts_tablet_manager.cc:1918] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.930612 1166 log.cc:1199] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3bed3f04c60249aaa389c26788974f58
I20250625 07:52:27.931123 1160 ts_tablet_manager.cc:1918] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.932122 666 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6d1017546c484c9a89ec039b91c63ad4
I20250625 07:52:27.931996 588 catalog_manager.cc:5582] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:27.932583 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:27.932961 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:27.934098 1166 ts_tablet_manager.cc:1939] T 3bed3f04c60249aaa389c26788974f58 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:27.934420 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_da048cffd3284b26bcbbba35f2a14300" } new_table_name: "default.Table_986822f6b2384bfdbbba4d39405a3f8b" modify_external_catalogs: true
I20250625 07:52:27.940994 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3bed3f04c60249aaa389c26788974f58 (table default.Table_a6db9072062d4e5895af0868d99ac31b [id=8e22a511cbc44f8996d35e514dfda631]) successfully deleted
I20250625 07:52:27.945144 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5a68ddb21adc44d584bf839cb84d1c18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.946422 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a68ddb21adc44d584bf839cb84d1c18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.948020 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fdcd25178a124f4fbe5c676be9adb5fc" } new_table_name: "default.Table_78878e8f1e8d4bb78aedb8e3e9a5c436" modify_external_catalogs: true
I20250625 07:52:27.931216 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.934793 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.937155 1163 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:27.948846 1160 log.cc:1199] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:27.948869 1163 ts_tablet_manager.cc:1397] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.019s
I20250625 07:52:27.949710 1163 raft_consensus.cc:357] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.949903 1163 raft_consensus.cc:383] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.949980 1163 raft_consensus.cc:738] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.950192 1163 consensus_queue.cc:260] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.950680 1163 ts_tablet_manager.cc:1428] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:27.950974 1163 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:27.951172 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4f1c150fb54c405c8dc61ac133d98b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1c150fb54c405c8dc61ac133d98b13 already in progress: creating tablet
W20250625 07:52:27.951965 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because the tablet was not found. No further retry: Not found: Tablet not found: 6d1017546c484c9a89ec039b91c63ad4
I20250625 07:52:27.952667 1163 tablet_bootstrap.cc:654] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.954072 1160 ts_tablet_manager.cc:1939] T 06e4bf78218f4b82b4f16f59b1704256 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:27.954588 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ba73f3114ea04e219b7a53b12922b3ea" } new_table_name: "default.Table_5a8a06415cdf4601a5b6749ad29a2a19" modify_external_catalogs: true
I20250625 07:52:27.954890 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 856) took 1481 ms. Trace:
I20250625 07:52:27.954984 1160 rpcz_store.cc:276] 0625 07:52:26.473277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.522773 (+ 49496us) service_pool.cc:225] Handling call
0625 07:52:27.862859 (+1340086us) ts_tablet_manager.cc:1145] Deleting tablet 06e4bf78218f4b82b4f16f59b1704256
0625 07:52:27.862876 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:27.896220 (+ 33344us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.931014 (+ 34794us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:27.954864 (+ 23850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339814}
I20250625 07:52:27.955374 1160 tablet_replica.cc:331] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:27.955550 1160 raft_consensus.cc:2241] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:27.955535 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 06e4bf78218f4b82b4f16f59b1704256 (table default.Table_4f95753979a34ea994409ba5978b7186 [id=4ff84ce52a874a5792b79f35f2016f1b]) successfully deleted
I20250625 07:52:27.955684 1160 raft_consensus.cc:2270] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:27.956419 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4e2dc27726d944c5885d62ed4258acfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.956943 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e2dc27726d944c5885d62ed4258acfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:27.959106 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f5b39a2075a487d8cbe4d4c9319f779" } new_table_name: "default.Table_dacbf51235d9416ab523aa0e65e0f8f6" modify_external_catalogs: true
W20250625 07:52:27.960707 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 32a2929bad2340db9e6e0b5364ec8567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a2929bad2340db9e6e0b5364ec8567 already in progress: creating tablet
I20250625 07:52:27.963069 1160 ts_tablet_manager.cc:1905] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:27.963584 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0bfcef152c9844a59989c532bd60c07d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bfcef152c9844a59989c532bd60c07d already in progress: creating tablet
I20250625 07:52:27.966914 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5ddc30bfdeea430e8109fea6979694a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:27.967459 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ddc30bfdeea430e8109fea6979694a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:27.969514 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 81451d3745f24cbd9a56307c557e5ec1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81451d3745f24cbd9a56307c557e5ec1 already in progress: creating tablet
I20250625 07:52:27.969969 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b48bc90f80f14bc2b35ce85acafa9028
W20250625 07:52:27.970999 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because the tablet was not found. No further retry: Not found: Tablet not found: 6d1017546c484c9a89ec039b91c63ad4
I20250625 07:52:27.971261 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:27.971755 747 tablet_service.cc:1515] Processing DeleteTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:27.972334 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b48bc90f80f14bc2b35ce85acafa9028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b48bc90f80f14bc2b35ce85acafa9028 already in progress: creating tablet
I20250625 07:52:27.904469 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.905453 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a2929bad2340db9e6e0b5364ec8567 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.907038 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.919265 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 81451d3745f24cbd9a56307c557e5ec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.925483 1158 tablet_bootstrap.cc:492] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:27.975661 1158 ts_tablet_manager.cc:1397] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.004s
I20250625 07:52:27.976590 1158 raft_consensus.cc:357] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.976832 1158 raft_consensus.cc:383] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:27.976897 1158 raft_consensus.cc:738] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:27.977147 1158 consensus_queue.cc:260] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:27.977743 1158 ts_tablet_manager.cc:1428] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:27.978097 1158 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:27.979537 747 tablet_service.cc:1468] Processing CreateTablet for tablet 94abb9c8a70a41169fb602c159d92f68 (DEFAULT_TABLE table=default.Table_8d4a225cda044dba96006acca0260845 [id=af6bb013e8f94342898d4b9ea1b61869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.980046 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abb9c8a70a41169fb602c159d92f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.980142 889 tablet_service.cc:1468] Processing CreateTablet for tablet 94abb9c8a70a41169fb602c159d92f68 (DEFAULT_TABLE table=default.Table_8d4a225cda044dba96006acca0260845 [id=af6bb013e8f94342898d4b9ea1b61869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.980487 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abb9c8a70a41169fb602c159d92f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.981245 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 2677e6bb2d474390990587e9e9887dfa (DEFAULT_TABLE table=default.Table_2c51d93cc45a4f19b57e86a55ed454a0 [id=8c92936a328b41b9ac4e8152fb07045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.983871 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2677e6bb2d474390990587e9e9887dfa (DEFAULT_TABLE table=default.Table_2c51d93cc45a4f19b57e86a55ed454a0 [id=8c92936a328b41b9ac4e8152fb07045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.984202 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2677e6bb2d474390990587e9e9887dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.934314 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.935438 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 94abb9c8a70a41169fb602c159d92f68 (DEFAULT_TABLE table=default.Table_8d4a225cda044dba96006acca0260845 [id=af6bb013e8f94342898d4b9ea1b61869]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.985570 1158 tablet_bootstrap.cc:654] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:27.938279 1161 ts_tablet_manager.cc:1918] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:27.986469 1161 log.cc:1199] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/de9ccb0a2d3846228acbc2769786a059
I20250625 07:52:27.986748 1161 ts_tablet_manager.cc:1939] T de9ccb0a2d3846228acbc2769786a059 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:27.970997 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet b48bc90f80f14bc2b35ce85acafa9028 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.991302 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2677e6bb2d474390990587e9e9887dfa (DEFAULT_TABLE table=default.Table_2c51d93cc45a4f19b57e86a55ed454a0 [id=8c92936a328b41b9ac4e8152fb07045b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.991804 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2677e6bb2d474390990587e9e9887dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.992247 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2677e6bb2d474390990587e9e9887dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.993016 1023 tablet_service.cc:1468] Processing CreateTablet for tablet dbdef039d3f842e5957c5bab85bedd42 (DEFAULT_TABLE table=default.Table_acf7718b3b9b41e2a0de6365460020fb [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.993561 754 tablet_service.cc:1468] Processing CreateTablet for tablet dbdef039d3f842e5957c5bab85bedd42 (DEFAULT_TABLE table=default.Table_acf7718b3b9b41e2a0de6365460020fb [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:27.993896 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdef039d3f842e5957c5bab85bedd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.997278 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdef039d3f842e5957c5bab85bedd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:27.998741 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e41e53efb6f342d2b1e4d642f7aaad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:27.999680 889 tablet_service.cc:1468] Processing CreateTablet for tablet dbdef039d3f842e5957c5bab85bedd42 (DEFAULT_TABLE table=default.Table_acf7718b3b9b41e2a0de6365460020fb [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.000082 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbdef039d3f842e5957c5bab85bedd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.000808 1856 raft_consensus.cc:491] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.000960 1856 raft_consensus.cc:513] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.001579 1856 leader_election.cc:290] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.001958 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 8a899790713d4646a850971bc3aebd9f (DEFAULT_TABLE table=default.Table_7485d0b61b2242e688d4b47cebfcb9cc [id=84a97784ccdf4d41bafb58f58f730ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.002286 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a899790713d4646a850971bc3aebd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.007440 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94abb9c8a70a41169fb602c159d92f68. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.007876 713 leader_election.cc:343] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7db110bb6cc146a393d62ddabbe1955a
I20250625 07:52:28.008849 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db110bb6cc146a393d62ddabbe1955a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:28.010231 713 leader_election.cc:343] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.010354 713 leader_election.cc:304] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.010710 1856 raft_consensus.cc:2747] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.011497 1160 ts_tablet_manager.cc:1918] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.011595 1160 log.cc:1199] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:28.011859 1160 ts_tablet_manager.cc:1939] T fc6c3be49be14ecbb3c26ef107ede015 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.013099 1161 tablet_replica.cc:331] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:28.013273 1161 raft_consensus.cc:2241] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.013302 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8a899790713d4646a850971bc3aebd9f (DEFAULT_TABLE table=default.Table_7485d0b61b2242e688d4b47cebfcb9cc [id=84a97784ccdf4d41bafb58f58f730ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.013661 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a899790713d4646a850971bc3aebd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.014350 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet de9ccb0a2d3846228acbc2769786a059 (table default.Table_66371f9b27d84da78425fe46ee94cdcf [id=6ec2407568244ae3988bff788a9c87d8]) successfully deleted
I20250625 07:52:28.014451 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 (DEFAULT_TABLE table=default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04 [id=8af32611ab2d4e1f83df1c4d4b243e22]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:28.014657 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5de0c5bf60b24808af124c95833d9c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de0c5bf60b24808af124c95833d9c25 already in progress: creating tablet
I20250625 07:52:28.014775 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1017546c484c9a89ec039b91c63ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.013414 1161 raft_consensus.cc:2270] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.015818 1161 ts_tablet_manager.cc:1905] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.021868 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 (DEFAULT_TABLE table=default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04 [id=8af32611ab2d4e1f83df1c4d4b243e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.022284 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1017546c484c9a89ec039b91c63ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.024999 746 tablet_service.cc:1468] Processing CreateTablet for tablet 73fed3e3517a48749af5b4a136cb9c02 (DEFAULT_TABLE table=default.Table_2b0d5eec0a7841b1bba234b5e57b41f3 [id=d48877f257e047f790e19213186abf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.025329 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fed3e3517a48749af5b4a136cb9c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.026842 754 tablet_service.cc:1468] Processing CreateTablet for tablet 99acd74c01f34d068c868fde3c6ec1df (DEFAULT_TABLE table=default.Table_bdc29c3cb4b44fdabdaa413f46834dcb [id=dd9e438abba847c3af80c0d0eee007cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.027140 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99acd74c01f34d068c868fde3c6ec1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.028911 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8366f344ec194ca2a23b5170ee3a5025 (DEFAULT_TABLE table=default.Table_329926f2f3e8457abd8548caa962293c [id=f193b97ed63b44efbcaaf9c88e33488a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.029243 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8366f344ec194ca2a23b5170ee3a5025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.031102 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dd10ce6555604cb084e1b1d299cf9aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.031678 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd10ce6555604cb084e1b1d299cf9aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.032958 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8d9333ac16747b48830505bb5f1c2d5 consensus state has no leader
I20250625 07:52:28.033409 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.033933 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.035601 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1633800aa3f442aa3a2d57538ab3ba9 consensus state has no leader
I20250625 07:52:28.035497 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_acf7718b3b9b41e2a0de6365460020fb" } new_table_name: "default.Table_42d926918dbf4b398f69aa2b3504f1b7" modify_external_catalogs: true
W20250625 07:52:28.035979 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 267f9accd9b34154aa0896055a930c73 consensus state has no leader
I20250625 07:52:28.036509 879 tablet_service.cc:1468] Processing CreateTablet for tablet 99acd74c01f34d068c868fde3c6ec1df (DEFAULT_TABLE table=default.Table_bdc29c3cb4b44fdabdaa413f46834dcb [id=dd9e438abba847c3af80c0d0eee007cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.036933 893 tablet_service.cc:1468] Processing CreateTablet for tablet d13d9c0065e34a96b6bdd29592932414 (DEFAULT_TABLE table=default.Table_125f22225a154f1288ab7309b9522d36 [id=e244ab96d2344361bf2d235670a5670f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.037042 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99acd74c01f34d068c868fde3c6ec1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.037294 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d9c0065e34a96b6bdd29592932414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.037448 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6d1017546c484c9a89ec039b91c63ad4 (DEFAULT_TABLE table=default.Table_47ea68a2b1fb4f4684bc9fc8dd74ce04 [id=8af32611ab2d4e1f83df1c4d4b243e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.037564 895 tablet_service.cc:1468] Processing CreateTablet for tablet 45855a33616e441b965dc8b14ab4b4bf (DEFAULT_TABLE table=default.Table_f5b05c08da4a4058ad3334ce9b0a8a84 [id=3fe65f183bbd4420a1e3f2bf983ae241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.037726 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d1017546c484c9a89ec039b91c63ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.037885 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45855a33616e441b965dc8b14ab4b4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.038168 883 tablet_service.cc:1468] Processing CreateTablet for tablet 73fed3e3517a48749af5b4a136cb9c02 (DEFAULT_TABLE table=default.Table_2b0d5eec0a7841b1bba234b5e57b41f3 [id=d48877f257e047f790e19213186abf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.038437 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fed3e3517a48749af5b4a136cb9c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.040920 876 tablet_service.cc:1468] Processing CreateTablet for tablet 5582df28a7d54fe8ad16b2f8ea1edabb (DEFAULT_TABLE table=default.Table_ddf70e7ccb3644f4b28ceb97decd744a [id=4a56615bca0a4d3bbb159007d1456583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.041105 888 tablet_service.cc:1468] Processing CreateTablet for tablet e01b29afb74147528807f30d8ccb8fe6 (DEFAULT_TABLE table=default.Table_51136d4037744ddfbd2b299213db9abd [id=531099c4dd364ca38470783b62ebd525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.041261 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5582df28a7d54fe8ad16b2f8ea1edabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.041324 1163 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:28.041445 1163 ts_tablet_manager.cc:1397] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.021s
I20250625 07:52:28.040920 877 tablet_service.cc:1468] Processing CreateTablet for tablet 8366f344ec194ca2a23b5170ee3a5025 (DEFAULT_TABLE table=default.Table_329926f2f3e8457abd8548caa962293c [id=f193b97ed63b44efbcaaf9c88e33488a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.042584 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8366f344ec194ca2a23b5170ee3a5025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.034994 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_5da3464e644f46ed9a0bd81795853dec [id=1ab8d615f8a84fdeba96d8266c1d4555] alter complete (version 3)
I20250625 07:52:28.043119 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01b29afb74147528807f30d8ccb8fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.044199 744 tablet_service.cc:1468] Processing CreateTablet for tablet 45855a33616e441b965dc8b14ab4b4bf (DEFAULT_TABLE table=default.Table_f5b05c08da4a4058ad3334ce9b0a8a84 [id=3fe65f183bbd4420a1e3f2bf983ae241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.044561 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45855a33616e441b965dc8b14ab4b4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.044906 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5932d19a0f33434f823db2c3060607fd (DEFAULT_TABLE table=default.Table_76ef6d59f0be4bc4804786565f912525 [id=25c682800cf5415593b9d1a2347b4d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.045161 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.045233 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5932d19a0f33434f823db2c3060607fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.045498 750 tablet_service.cc:1468] Processing CreateTablet for tablet d13d9c0065e34a96b6bdd29592932414 (DEFAULT_TABLE table=default.Table_125f22225a154f1288ab7309b9522d36 [id=e244ab96d2344361bf2d235670a5670f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.045778 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d9c0065e34a96b6bdd29592932414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.046089 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e01b29afb74147528807f30d8ccb8fe6 (DEFAULT_TABLE table=default.Table_51136d4037744ddfbd2b299213db9abd [id=531099c4dd364ca38470783b62ebd525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.046584 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5932d19a0f33434f823db2c3060607fd (DEFAULT_TABLE table=default.Table_76ef6d59f0be4bc4804786565f912525 [id=25c682800cf5415593b9d1a2347b4d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.047034 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d13d9c0065e34a96b6bdd29592932414 (DEFAULT_TABLE table=default.Table_125f22225a154f1288ab7309b9522d36 [id=e244ab96d2344361bf2d235670a5670f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.048929 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 5582df28a7d54fe8ad16b2f8ea1edabb (DEFAULT_TABLE table=default.Table_ddf70e7ccb3644f4b28ceb97decd744a [id=4a56615bca0a4d3bbb159007d1456583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.048909 756 tablet_service.cc:1468] Processing CreateTablet for tablet e68b52183b3f4b11b81d7cee20391953 (DEFAULT_TABLE table=default.Table_dc7e4a1ebfbe4239a253f502af55d599 [id=9c5f122b5ce84a87a4fd67292f9f6e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.049119 745 tablet_service.cc:1468] Processing CreateTablet for tablet 059dad944716497bbba17e11fb3ba5be (DEFAULT_TABLE table=default.Table_f260806f3d3d4af59e7c35222a769938 [id=01e5a3e22aff4a9bb0feffce780f37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.049249 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68b52183b3f4b11b81d7cee20391953. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.049399 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059dad944716497bbba17e11fb3ba5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.049419 1013 tablet_service.cc:1468] Processing CreateTablet for tablet e68b52183b3f4b11b81d7cee20391953 (DEFAULT_TABLE table=default.Table_dc7e4a1ebfbe4239a253f502af55d599 [id=9c5f122b5ce84a87a4fd67292f9f6e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.049664 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1d4be984846344bbb20a1076112b4330 (DEFAULT_TABLE table=default.Table_f1f321543e7342108ad8b10b3c846822 [id=23c51eaec33f4548a1cbd9a078631b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.044566 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.050168 741 tablet_service.cc:1468] Processing CreateTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 (DEFAULT_TABLE table=default.Table_b1873a0532e24b77925f3c5b029dcda3 [id=783e56f2ba7c421ab8acf84f450ff8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.050326 751 tablet_service.cc:1468] Processing CreateTablet for tablet 5582df28a7d54fe8ad16b2f8ea1edabb (DEFAULT_TABLE table=default.Table_ddf70e7ccb3644f4b28ceb97decd744a [id=4a56615bca0a4d3bbb159007d1456583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.050861 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 (DEFAULT_TABLE table=default.Table_ecd308318d99472b929b8e811d4d613f [id=def1f77e10ce4f03a3490a687184a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.051246 739 tablet_service.cc:1468] Processing CreateTablet for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9 (DEFAULT_TABLE table=default.Table_4e7e572213414b8aa45a1abf1915a26d [id=3b0e52a394fa47649dc1e6482e4693a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.048935 748 tablet_service.cc:1468] Processing CreateTablet for tablet bf06bd55d69c4c118cf15006eba33fb4 (DEFAULT_TABLE table=default.Table_12bc5fb366f644f1a5566b904a5c5ba9 [id=a74dea902d93437d9a9840059d9beb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.049938 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4be984846344bbb20a1076112b4330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.053009 1016 tablet_service.cc:1468] Processing CreateTablet for tablet bf06bd55d69c4c118cf15006eba33fb4 (DEFAULT_TABLE table=default.Table_12bc5fb366f644f1a5566b904a5c5ba9 [id=a74dea902d93437d9a9840059d9beb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.053097 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.053534 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf06bd55d69c4c118cf15006eba33fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.053478 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 059dad944716497bbba17e11fb3ba5be (DEFAULT_TABLE table=default.Table_f260806f3d3d4af59e7c35222a769938 [id=01e5a3e22aff4a9bb0feffce780f37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.053894 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 1d4be984846344bbb20a1076112b4330 (DEFAULT_TABLE table=default.Table_f1f321543e7342108ad8b10b3c846822 [id=23c51eaec33f4548a1cbd9a078631b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.053968 758 tablet_service.cc:1468] Processing CreateTablet for tablet e01b29afb74147528807f30d8ccb8fe6 (DEFAULT_TABLE table=default.Table_51136d4037744ddfbd2b299213db9abd [id=531099c4dd364ca38470783b62ebd525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.054327 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 (DEFAULT_TABLE table=default.Table_2185371a6bd1488b88a8038243706d02 [id=86edf45af74844b3982f423805c3983c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.054436 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5582df28a7d54fe8ad16b2f8ea1edabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.053011 752 tablet_service.cc:1468] Processing CreateTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 (DEFAULT_TABLE table=default.Table_2185371a6bd1488b88a8038243706d02 [id=86edf45af74844b3982f423805c3983c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.054777 1017 tablet_service.cc:1468] Processing CreateTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 (DEFAULT_TABLE table=default.Table_b1873a0532e24b77925f3c5b029dcda3 [id=783e56f2ba7c421ab8acf84f450ff8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.055246 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 (DEFAULT_TABLE table=default.Table_ecd308318d99472b929b8e811d4d613f [id=def1f77e10ce4f03a3490a687184a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.055876 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be8ab5140aa4d48be978519cb99ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.055872 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9 (DEFAULT_TABLE table=default.Table_4e7e572213414b8aa45a1abf1915a26d [id=3b0e52a394fa47649dc1e6482e4693a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.056401 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3ccad4a5a34bfba36c5c18ec5038c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.056628 1025 tablet_service.cc:1468] Processing CreateTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 (DEFAULT_TABLE table=default.Table_7b26265e9f9a44bcbd5ba18263f61864 [id=6308311f525145068708c8530fd1c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.056407 1856 raft_consensus.cc:491] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.056846 1856 raft_consensus.cc:513] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:28.057343 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d38786a3d3a9450bb6efde1675302683 consensus state has no leader
I20250625 07:52:28.057451 1856 leader_election.cc:290] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.058184 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8a899790713d4646a850971bc3aebd9f (DEFAULT_TABLE table=default.Table_7485d0b61b2242e688d4b47cebfcb9cc [id=84a97784ccdf4d41bafb58f58f730ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.058885 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a899790713d4646a850971bc3aebd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.064428 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 73fed3e3517a48749af5b4a136cb9c02 (DEFAULT_TABLE table=default.Table_2b0d5eec0a7841b1bba234b5e57b41f3 [id=d48877f257e047f790e19213186abf30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.065445 892 tablet_service.cc:1468] Processing CreateTablet for tablet 5932d19a0f33434f823db2c3060607fd (DEFAULT_TABLE table=default.Table_76ef6d59f0be4bc4804786565f912525 [id=25c682800cf5415593b9d1a2347b4d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.065636 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01b29afb74147528807f30d8ccb8fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.065845 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5932d19a0f33434f823db2c3060607fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.065865 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5582df28a7d54fe8ad16b2f8ea1edabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.066035 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4be984846344bbb20a1076112b4330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.068802 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e6a2f10bc14896b6bc01bdc0da3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.071316 1158 tablet_bootstrap.cc:492] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:28.071455 1158 ts_tablet_manager.cc:1397] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.003s
I20250625 07:52:28.072055 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68b52183b3f4b11b81d7cee20391953. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.072314 1158 raft_consensus.cc:357] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.072479 1158 raft_consensus.cc:383] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.072526 1158 raft_consensus.cc:738] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.072773 1158 consensus_queue.cc:260] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.072883 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf06bd55d69c4c118cf15006eba33fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.073374 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be8ab5140aa4d48be978519cb99ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.075330 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01b29afb74147528807f30d8ccb8fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.075793 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e6a2f10bc14896b6bc01bdc0da3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.079380 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98c1f91e4914fb29c6b413e92cb1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.079773 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73fed3e3517a48749af5b4a136cb9c02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.080842 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3ccad4a5a34bfba36c5c18ec5038c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.081293 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13d9c0065e34a96b6bdd29592932414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.083141 1158 ts_tablet_manager.cc:1428] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250625 07:52:28.083449 1158 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:28.084458 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059dad944716497bbba17e11fb3ba5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.084981 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.085847 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5932d19a0f33434f823db2c3060607fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.090600 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 863) took 1503 ms. Trace:
I20250625 07:52:28.090747 1160 rpcz_store.cc:276] 0625 07:52:26.586256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.586990 (+ 734us) service_pool.cc:225] Handling call
0625 07:52:27.955329 (+1368339us) ts_tablet_manager.cc:1145] Deleting tablet fc6c3be49be14ecbb3c26ef107ede015
0625 07:52:27.955339 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.008437 (+ 53098us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.011410 (+ 2973us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.089470 (+ 78060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368040}
I20250625 07:52:28.097501 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3613c8cb799746c294a74ca856714124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.098138 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3613c8cb799746c294a74ca856714124 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.099848 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 99acd74c01f34d068c868fde3c6ec1df (DEFAULT_TABLE table=default.Table_bdc29c3cb4b44fdabdaa413f46834dcb [id=dd9e438abba847c3af80c0d0eee007cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.100203 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99acd74c01f34d068c868fde3c6ec1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.100533 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c3197b8d0402490b86e0fbc10c91c62e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.101066 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3197b8d0402490b86e0fbc10c91c62e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.091645 1163 raft_consensus.cc:357] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.100502 1925 raft_consensus.cc:491] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.101469 1163 raft_consensus.cc:383] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.101539 1163 raft_consensus.cc:738] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.101503 1925 raft_consensus.cc:513] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.101732 1163 consensus_queue.cc:260] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.102123 1925 leader_election.cc:290] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.102171 1163 ts_tablet_manager.cc:1428] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.061s user 0.001s sys 0.001s
I20250625 07:52:28.102435 1163 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:28.104163 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd728ffc2a274deea3fad533d1eaa6da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:28.104712 851 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.105237 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 865) took 1470 ms. Trace:
I20250625 07:52:28.105378 1160 rpcz_store.cc:276] 0625 07:52:26.635062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.635118 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:28.103525 (+1468407us) ts_tablet_manager.cc:1145] Deleting tablet 8f09c9fd1b3a4dae9c8b366000737b5e
0625 07:52:28.103724 (+ 199us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.105221 (+ 1497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468147}
I20250625 07:52:28.105756 1160 tablet_replica.cc:331] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.105959 1160 raft_consensus.cc:2241] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.106380 1160 raft_consensus.cc:2270] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.106477 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 8366f344ec194ca2a23b5170ee3a5025 (DEFAULT_TABLE table=default.Table_329926f2f3e8457abd8548caa962293c [id=f193b97ed63b44efbcaaf9c88e33488a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.106992 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8366f344ec194ca2a23b5170ee3a5025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.107216 1160 ts_tablet_manager.cc:1905] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.107381 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 45855a33616e441b965dc8b14ab4b4bf (DEFAULT_TABLE table=default.Table_f5b05c08da4a4058ad3334ce9b0a8a84 [id=3fe65f183bbd4420a1e3f2bf983ae241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.107684 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45855a33616e441b965dc8b14ab4b4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.109292 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c6fe509bff964fc783b30da1badc3431" } modify_external_catalogs: true
I20250625 07:52:28.109474 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c6fe509bff964fc783b30da1badc3431" } modify_external_catalogs: true
I20250625 07:52:28.115911 1160 ts_tablet_manager.cc:1918] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.120394 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.121398 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2185371a6bd1488b88a8038243706d02" } modify_external_catalogs: true
I20250625 07:52:28.121587 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2185371a6bd1488b88a8038243706d02" } modify_external_catalogs: true
I20250625 07:52:28.124667 1160 log.cc:1199] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:28.125315 1162 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.125535 1162 ts_tablet_manager.cc:1397] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.228s user 0.006s sys 0.005s
I20250625 07:52:28.126658 1162 raft_consensus.cc:357] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.126880 1162 raft_consensus.cc:383] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.127009 1162 raft_consensus.cc:738] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.127290 1162 consensus_queue.cc:260] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.128057 1162 ts_tablet_manager.cc:1428] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:28.129230 1162 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.130686 754 tablet_service.cc:1468] Processing CreateTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 (DEFAULT_TABLE table=default.Table_7b26265e9f9a44bcbd5ba18263f61864 [id=6308311f525145068708c8530fd1c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.131027 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98c1f91e4914fb29c6b413e92cb1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.131817 1162 tablet_bootstrap.cc:654] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.132611 755 tablet_service.cc:1468] Processing CreateTablet for tablet 33f56c267457451e864e277e110c68fa (DEFAULT_TABLE table=default.Table_a559f40e04b9464ab39d21bf2fdd6ac6 [id=37ff1c3f86d0463a8795ac1ea6ef45ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.132965 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f56c267457451e864e277e110c68fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.139454 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2166d8d523c34441943a1d577b745b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.139753 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f39da4a74c09429db8be207816d8f5d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.140048 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2166d8d523c34441943a1d577b745b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.140244 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f39da4a74c09429db8be207816d8f5d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.140738 758 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.123314 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c0bbd21d4aa41b2b92473bffe7dc05a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.143481 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_96d96b9fee4d48efaeb2f22ff67b43e6" } modify_external_catalogs: true
I20250625 07:52:28.143703 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_96d96b9fee4d48efaeb2f22ff67b43e6" } modify_external_catalogs: true
W20250625 07:52:28.145275 713 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.146859 1162 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.147030 1162 ts_tablet_manager.cc:1397] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20250625 07:52:28.147861 1162 raft_consensus.cc:357] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.148090 1162 raft_consensus.cc:383] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.148170 1162 raft_consensus.cc:738] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.148422 1162 consensus_queue.cc:260] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.149212 1162 ts_tablet_manager.cc:1428] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:28.149505 1162 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.151082 1162 tablet_bootstrap.cc:654] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.158288 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 33f56c267457451e864e277e110c68fa (DEFAULT_TABLE table=default.Table_a559f40e04b9464ab39d21bf2fdd6ac6 [id=37ff1c3f86d0463a8795ac1ea6ef45ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.158778 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f56c267457451e864e277e110c68fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.159801 1162 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.159994 1162 ts_tablet_manager.cc:1397] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20250625 07:52:28.160789 1162 raft_consensus.cc:357] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.161023 1162 raft_consensus.cc:383] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.161113 1162 raft_consensus.cc:738] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.161383 1162 consensus_queue.cc:260] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.162117 1162 ts_tablet_manager.cc:1428] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:28.162433 1733 raft_consensus.cc:491] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.162592 1162 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.165282 1162 tablet_bootstrap.cc:654] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.162611 1733 raft_consensus.cc:513] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.166203 1733 leader_election.cc:290] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:28.169791 986 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b377c8111c9f4ac2ba05e65694d611b9
W20250625 07:52:28.172595 985 leader_election.cc:343] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b377c8111c9f4ac2ba05e65694d611b9
I20250625 07:52:28.172812 985 leader_election.cc:304] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:28.173730 1166 tablet_replica.cc:331] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.174083 1166 raft_consensus.cc:2241] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.174304 1166 raft_consensus.cc:2270] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.175568 1162 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.175767 1162 ts_tablet_manager.cc:1397] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250625 07:52:28.176614 1158 tablet_bootstrap.cc:654] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.176528 1162 raft_consensus.cc:357] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.176883 1162 raft_consensus.cc:383] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.176996 1162 raft_consensus.cc:738] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.177271 1162 consensus_queue.cc:260] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.173391 1733 raft_consensus.cc:2747] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.178239 1162 ts_tablet_manager.cc:1428] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:28.178531 1162 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.180245 1162 tablet_bootstrap.cc:654] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.180868 1166 ts_tablet_manager.cc:1905] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.193696 1166 ts_tablet_manager.cc:1918] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.193930 1166 log.cc:1199] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6239713a586d43c28c101ba2447d579d
I20250625 07:52:28.194285 1166 ts_tablet_manager.cc:1939] T 6239713a586d43c28c101ba2447d579d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.195217 1166 tablet_replica.cc:331] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.195544 1166 raft_consensus.cc:2241] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.195716 1166 raft_consensus.cc:2270] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.196460 1166 ts_tablet_manager.cc:1905] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.202359 1166 ts_tablet_manager.cc:1918] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.202540 1166 log.cc:1199] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:28.202910 1166 ts_tablet_manager.cc:1939] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.203513 1166 tablet_replica.cc:331] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.203554 1856 raft_consensus.cc:491] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.203732 1166 raft_consensus.cc:2241] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.203694 1856 raft_consensus.cc:513] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.203866 1166 raft_consensus.cc:2270] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.204283 1856 leader_election.cc:290] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.204386 1166 ts_tablet_manager.cc:1905] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:28.208635 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbdef039d3f842e5957c5bab85bedd42 consensus state has no leader
I20250625 07:52:28.211784 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cebbbf28857c46a6adc44443c9a09488" } modify_external_catalogs: true
I20250625 07:52:28.211987 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cebbbf28857c46a6adc44443c9a09488" } modify_external_catalogs: true
I20250625 07:52:28.212253 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cb1093429f4646c7ade95e18696a76bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.212903 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb1093429f4646c7ade95e18696a76bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.214424 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_014505d5378f4c5a89fefb3ada354601" } modify_external_catalogs: true
I20250625 07:52:28.214612 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_014505d5378f4c5a89fefb3ada354601" } modify_external_catalogs: true
I20250625 07:52:28.216050 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 0f566cc9cdd447798a5cb408ef9cd35a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:28.216912 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a3e6a2f10bc14896b6bc01bdc0da3630
I20250625 07:52:28.218114 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f1983df4902f48d2b408a99b440368e7
I20250625 07:52:28.219328 1166 ts_tablet_manager.cc:1918] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.219427 1166 log.cc:1199] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7db110bb6cc146a393d62ddabbe1955a
I20250625 07:52:28.219687 1166 ts_tablet_manager.cc:1939] T 7db110bb6cc146a393d62ddabbe1955a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.222327 1161 ts_tablet_manager.cc:1918] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.222445 1161 log.cc:1199] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c7a2b94af59648239cfd1e5ce5e8b968
I20250625 07:52:28.222702 1161 ts_tablet_manager.cc:1939] T c7a2b94af59648239cfd1e5ce5e8b968 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:28.223688 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b47065e0382b4b9a936613a0ad84695d (DEFAULT_TABLE table=default.Table_77d400c9d724449c8fb62b8761fb2784 [id=d23fcd6e6f0a418687db7d4dc4862974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.224067 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47065e0382b4b9a936613a0ad84695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.225219 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 78f0921af91140c6989de2306326c28f
I20250625 07:52:28.234490 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6a808c6d46a94091b18c21073a499b8a" } new_table_name: "default.Table_3638af2256ed45d68ee8334b9b3c30b4" modify_external_catalogs: true
I20250625 07:52:28.236097 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f5ab0400081848dd872a429685e3b8e2
I20250625 07:52:28.236163 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 08909560b69d4171a7583e293c661632 (DEFAULT_TABLE table=default.Table_99eddf6f5e75402fb97ac6cc3375e188 [id=c5886d86d74d415fbc39c056e4eac6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.236547 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08909560b69d4171a7583e293c661632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.236915 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c (DEFAULT_TABLE table=default.Table_146999741edf4bde91a875f03e40a74e [id=dce7233d9b9b431db7c58cfe5801d73e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.237300 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.237706 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b7c5c675d0a7470e95a30a499ad15714
I20250625 07:52:28.240921 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 (DEFAULT_TABLE table=default.Table_cfb63fdad31e4cada3c0e785a501be1d [id=d9803bb40f8941d990d04736f525e8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.241310 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d8f0eabf1410fadaadb98344b3f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.242074 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.243722 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1c370799a55b474080d93cb980cf5f39" } new_table_name: "default.Table_ca90d59a48d947778cd899359e52ed41" modify_external_catalogs: true
I20250625 07:52:28.243780 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3245e577c1b349f0b11133c6d1363526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.244297 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3245e577c1b349f0b11133c6d1363526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.245088 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c0c5f7be7e344a1c9172a48d12556733" } new_table_name: "default.Table_be2cfecee07e48e08dd59c09ada221cc" modify_external_catalogs: true
I20250625 07:52:28.246520 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_00fc5f49db204824b892b2ae69796dca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.247071 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00fc5f49db204824b892b2ae69796dca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.247534 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b527d528908b40de9346966341c2f3fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.247988 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b527d528908b40de9346966341c2f3fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.249112 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6239713a586d43c28c101ba2447d579d (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
I20250625 07:52:28.249168 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ca0144e3c5142ac9fcbd927b7585a3c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.249380 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
I20250625 07:52:28.249692 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c7a2b94af59648239cfd1e5ce5e8b968 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]) successfully deleted
I20250625 07:52:28.249799 740 tablet_service.cc:1468] Processing CreateTablet for tablet b47065e0382b4b9a936613a0ad84695d (DEFAULT_TABLE table=default.Table_77d400c9d724449c8fb62b8761fb2784 [id=d23fcd6e6f0a418687db7d4dc4862974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.250201 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47065e0382b4b9a936613a0ad84695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.250593 1011 tablet_service.cc:1468] Processing CreateTablet for tablet c1633800aa3f442aa3a2d57538ab3ba9 (DEFAULT_TABLE table=default.Table_78878e8f1e8d4bb78aedb8e3e9a5c436 [id=38e6614e81e24dca9c9bd0eeebac3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.250926 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1633800aa3f442aa3a2d57538ab3ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.251130 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet f5ab0400081848dd872a429685e3b8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.251327 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c5c675d0a7470e95a30a499ad15714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.253113 1161 tablet_replica.cc:331] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:28.253274 1161 raft_consensus.cc:2241] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.253404 1161 raft_consensus.cc:2270] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.253862 1161 ts_tablet_manager.cc:1905] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.254611 757 tablet_service.cc:1468] Processing CreateTablet for tablet c1633800aa3f442aa3a2d57538ab3ba9 (DEFAULT_TABLE table=default.Table_78878e8f1e8d4bb78aedb8e3e9a5c436 [id=38e6614e81e24dca9c9bd0eeebac3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.254918 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1633800aa3f442aa3a2d57538ab3ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.256783 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f1983df4902f48d2b408a99b440368e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.256920 756 tablet_service.cc:1468] Processing CreateTablet for tablet 08909560b69d4171a7583e293c661632 (DEFAULT_TABLE table=default.Table_99eddf6f5e75402fb97ac6cc3375e188 [id=c5886d86d74d415fbc39c056e4eac6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.257149 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f1983df4902f48d2b408a99b440368e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.257263 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08909560b69d4171a7583e293c661632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.257416 1166 tablet_replica.cc:331] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.257577 1166 raft_consensus.cc:2241] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.257624 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 78f0921af91140c6989de2306326c28f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.257704 1166 raft_consensus.cc:2270] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.258023 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f5ab0400081848dd872a429685e3b8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.258312 1166 ts_tablet_manager.cc:1905] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.258857 741 tablet_service.cc:1468] Processing CreateTablet for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c (DEFAULT_TABLE table=default.Table_146999741edf4bde91a875f03e40a74e [id=dce7233d9b9b431db7c58cfe5801d73e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.259049 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d3a57960634c41bd8b2bb81061f1a4ab" } new_table_name: "default.Table_5771064873114b4786e35a9a764c2bfd" modify_external_catalogs: true
I20250625 07:52:28.259209 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.260465 739 tablet_service.cc:1468] Processing CreateTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 (DEFAULT_TABLE table=default.Table_cfb63fdad31e4cada3c0e785a501be1d [id=d9803bb40f8941d990d04736f525e8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.260845 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.261160 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d8f0eabf1410fadaadb98344b3f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.261258 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 78f0921af91140c6989de2306326c28f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.262231 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c8c015c202544914b74f1e245b496f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.265350 751 tablet_service.cc:1468] Processing CreateTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 (DEFAULT_TABLE table=default.Table_ff871a8a9993429da4d087d0362c3ced [id=2ccb233f0c6c41f1a540d0cc9e5c1a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.265686 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee3980ed9747af87329fd3a1dfd699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.265671 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 (DEFAULT_TABLE table=default.Table_ff871a8a9993429da4d087d0362c3ced [id=2ccb233f0c6c41f1a540d0cc9e5c1a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.266008 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee3980ed9747af87329fd3a1dfd699. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.266324 713 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.266803 1925 raft_consensus.cc:491] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.266956 1925 raft_consensus.cc:513] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.267324 752 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c5c675d0a7470e95a30a499ad15714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.269006 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d (DEFAULT_TABLE table=default.Table_2b2ab9e4f189492dbfec4ec5c79a1630 [id=8e956b03bafc457b9d27d9fbbc21ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.269519 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f6f8c1154fc89b3aa1874441b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.270530 1160 ts_tablet_manager.cc:1939] T ba0735d576f947d395bee8d1b865f2fd P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.271950 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 27768205b8e24a09a2b0e02153781fcf (DEFAULT_TABLE table=default.Table_d02d6575202148d0bad916a7cba0191d [id=dd78e1d773e84f50904fef5c19cf6f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.272562 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27768205b8e24a09a2b0e02153781fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.273021 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7db110bb6cc146a393d62ddabbe1955a (table default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]) successfully deleted
I20250625 07:52:28.271360 876 tablet_service.cc:1468] Processing CreateTablet for tablet e68b52183b3f4b11b81d7cee20391953 (DEFAULT_TABLE table=default.Table_dc7e4a1ebfbe4239a253f502af55d599 [id=9c5f122b5ce84a87a4fd67292f9f6e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.271360 879 tablet_service.cc:1468] Processing CreateTablet for tablet 059dad944716497bbba17e11fb3ba5be (DEFAULT_TABLE table=default.Table_f260806f3d3d4af59e7c35222a769938 [id=01e5a3e22aff4a9bb0feffce780f37fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.272214 1925 leader_election.cc:290] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.272583 892 tablet_service.cc:1468] Processing CreateTablet for tablet bf06bd55d69c4c118cf15006eba33fb4 (DEFAULT_TABLE table=default.Table_12bc5fb366f644f1a5566b904a5c5ba9 [id=a74dea902d93437d9a9840059d9beb02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.272634 877 tablet_service.cc:1468] Processing CreateTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 (DEFAULT_TABLE table=default.Table_2185371a6bd1488b88a8038243706d02 [id=86edf45af74844b3982f423805c3983c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.272888 889 tablet_service.cc:1468] Processing CreateTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 (DEFAULT_TABLE table=default.Table_ecd308318d99472b929b8e811d4d613f [id=def1f77e10ce4f03a3490a687184a1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.272995 883 tablet_service.cc:1468] Processing CreateTablet for tablet 1d4be984846344bbb20a1076112b4330 (DEFAULT_TABLE table=default.Table_f1f321543e7342108ad8b10b3c846822 [id=23c51eaec33f4548a1cbd9a078631b1c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:28.275393 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d43d98c0c51b40c6bdb0c2705465038e consensus state has no leader
I20250625 07:52:28.276472 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68b52183b3f4b11b81d7cee20391953. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.276777 895 tablet_service.cc:1468] Processing CreateTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 (DEFAULT_TABLE table=default.Table_b1873a0532e24b77925f3c5b029dcda3 [id=783e56f2ba7c421ab8acf84f450ff8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.276865 891 tablet_service.cc:1468] Processing CreateTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 (DEFAULT_TABLE table=default.Table_7b26265e9f9a44bcbd5ba18263f61864 [id=6308311f525145068708c8530fd1c733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.277607 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b98c1f91e4914fb29c6b413e92cb1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.278435 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c (DEFAULT_TABLE table=default.Table_146999741edf4bde91a875f03e40a74e [id=dce7233d9b9b431db7c58cfe5801d73e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.278745 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f297fdcad1449c1a4cb6a5984bb5a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.279011 886 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c5c675d0a7470e95a30a499ad15714 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.279102 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a3e6a2f10bc14896b6bc01bdc0da3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.279309 881 tablet_service.cc:1468] Processing CreateTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 (DEFAULT_TABLE table=default.Table_ff871a8a9993429da4d087d0362c3ced [id=2ccb233f0c6c41f1a540d0cc9e5c1a6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.279282 886 tablet_service.cc:1468] Processing CreateTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 (DEFAULT_TABLE table=default.Table_cfb63fdad31e4cada3c0e785a501be1d [id=d9803bb40f8941d990d04736f525e8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.279615 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ee3980ed9747af87329fd3a1dfd699. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.279623 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9 (DEFAULT_TABLE table=default.Table_4e7e572213414b8aa45a1abf1915a26d [id=3b0e52a394fa47649dc1e6482e4693a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.279767 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059dad944716497bbba17e11fb3ba5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.279911 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b74d8f0eabf1410fadaadb98344b3f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.279935 884 tablet_service.cc:1515] Processing DeleteTablet for tablet f5ab0400081848dd872a429685e3b8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.280105 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf06bd55d69c4c118cf15006eba33fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.280246 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe3ccad4a5a34bfba36c5c18ec5038c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.280401 888 tablet_service.cc:1468] Processing CreateTablet for tablet 33f56c267457451e864e277e110c68fa (DEFAULT_TABLE table=default.Table_a559f40e04b9464ab39d21bf2fdd6ac6 [id=37ff1c3f86d0463a8795ac1ea6ef45ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.280622 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7be8ab5140aa4d48be978519cb99ed31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.280824 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0e17b618cd4c2983e22b3e1e6ecba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.281754 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d4be984846344bbb20a1076112b4330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.283782 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f56c267457451e864e277e110c68fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.285509 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 67ea9c5fac904443a0e6004f8bebedbd (DEFAULT_TABLE table=default.Table_1726e9aaa49544cf98ad7271dd414da8 [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.285859 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ea9c5fac904443a0e6004f8bebedbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.288010 1158 tablet_bootstrap.cc:492] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:28.288197 1158 ts_tablet_manager.cc:1397] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.205s user 0.006s sys 0.000s
I20250625 07:52:28.289266 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e6a2f10bc14896b6bc01bdc0da3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.289718 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d (DEFAULT_TABLE table=default.Table_2b2ab9e4f189492dbfec4ec5c79a1630 [id=8e956b03bafc457b9d27d9fbbc21ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.290026 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f6f8c1154fc89b3aa1874441b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.292412 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet fc6c3be49be14ecbb3c26ef107ede015 (table default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]) successfully deleted
I20250625 07:52:28.292997 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd728ffc2a274deea3fad533d1eaa6da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.298092 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 78f0921af91140c6989de2306326c28f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.298157 887 tablet_service.cc:1468] Processing CreateTablet for tablet c1633800aa3f442aa3a2d57538ab3ba9 (DEFAULT_TABLE table=default.Table_78878e8f1e8d4bb78aedb8e3e9a5c436 [id=38e6614e81e24dca9c9bd0eeebac3318]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.298238 880 tablet_service.cc:1515] Processing DeleteTablet for tablet f1983df4902f48d2b408a99b440368e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.298486 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1633800aa3f442aa3a2d57538ab3ba9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.299278 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8f09c9fd1b3a4dae9c8b366000737b5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f09c9fd1b3a4dae9c8b366000737b5e already in progress: creating tablet
I20250625 07:52:28.299437 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.300360 1158 raft_consensus.cc:357] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.300621 1158 raft_consensus.cc:383] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.300743 1158 raft_consensus.cc:738] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.301012 1158 consensus_queue.cc:260] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.304890 1158 ts_tablet_manager.cc:1428] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 07:52:28.307722 1158 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:28.309185 894 tablet_service.cc:1468] Processing CreateTablet for tablet 08909560b69d4171a7583e293c661632 (DEFAULT_TABLE table=default.Table_99eddf6f5e75402fb97ac6cc3375e188 [id=c5886d86d74d415fbc39c056e4eac6c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:28.312574 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7ed91509ba94562a9207c7d7c0d3e2b consensus state has no leader
W20250625 07:52:28.313235 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c3ea1daff2b4d15874c00ff075be8be consensus state has no leader
W20250625 07:52:28.314729 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 126e3c3c0b5e41599e7027708fefd7e7 consensus state has no leader
I20250625 07:52:28.317298 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08909560b69d4171a7583e293c661632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.319453 1163 tablet_bootstrap.cc:654] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.322465 882 tablet_service.cc:1468] Processing CreateTablet for tablet b47065e0382b4b9a936613a0ad84695d (DEFAULT_TABLE table=default.Table_77d400c9d724449c8fb62b8761fb2784 [id=d23fcd6e6f0a418687db7d4dc4862974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.322826 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47065e0382b4b9a936613a0ad84695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.328318 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:28.329233 851 leader_election.cc:343] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.332363 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.332814 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.333426 741 tablet_service.cc:1468] Processing CreateTablet for tablet 27768205b8e24a09a2b0e02153781fcf (DEFAULT_TABLE table=default.Table_d02d6575202148d0bad916a7cba0191d [id=dd78e1d773e84f50904fef5c19cf6f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.333832 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27768205b8e24a09a2b0e02153781fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.334470 754 tablet_service.cc:1468] Processing CreateTablet for tablet 67ea9c5fac904443a0e6004f8bebedbd (DEFAULT_TABLE table=default.Table_1726e9aaa49544cf98ad7271dd414da8 [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.335646 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ea9c5fac904443a0e6004f8bebedbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.326649 752 tablet_service.cc:1468] Processing CreateTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d (DEFAULT_TABLE table=default.Table_2b2ab9e4f189492dbfec4ec5c79a1630 [id=8e956b03bafc457b9d27d9fbbc21ee15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.335952 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4243f6f8c1154fc89b3aa1874441b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.346606 1925 raft_consensus.cc:491] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:28.346764 713 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:28.346755 1925 raft_consensus.cc:513] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.346868 713 leader_election.cc:304] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:28.347018 713 raft_consensus.cc:2684] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:28.347620 713 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:28.347729 713 leader_election.cc:304] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.347954 1856 raft_consensus.cc:2747] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:28.350344 850 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.350466 850 leader_election.cc:304] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.350683 1793 raft_consensus.cc:2747] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:28.351052 850 leader_election.cc:343] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.351158 850 leader_election.cc:304] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.351338 1793 raft_consensus.cc:2747] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.351959 1925 leader_election.cc:290] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.354255 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ab38e0cdcd14f209e6877f2e4a46e98" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.356583 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ab38e0cdcd14f209e6877f2e4a46e98" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:28.357653 851 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:28.358961 850 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.359068 850 leader_election.cc:304] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.359282 1925 raft_consensus.cc:2747] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.359252 1856 consensus_queue.cc:1035] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:28.381841 1166 ts_tablet_manager.cc:1918] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.382035 1166 log.cc:1199] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/fcc3afc210a540058294121d2b83c66d
I20250625 07:52:28.382393 1166 ts_tablet_manager.cc:1939] T fcc3afc210a540058294121d2b83c66d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.384121 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet fcc3afc210a540058294121d2b83c66d (table default.Table_bb831997b8cf4b99a7779a4a6b23a238 [id=8a01186fa596413fb75529d0d57a8d9d]) successfully deleted
I20250625 07:52:28.385722 895 tablet_service.cc:1468] Processing CreateTablet for tablet 67ea9c5fac904443a0e6004f8bebedbd (DEFAULT_TABLE table=default.Table_1726e9aaa49544cf98ad7271dd414da8 [id=63d51352cf0b435ba49ef141c86bc32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.385830 1162 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.386018 1162 ts_tablet_manager.cc:1397] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.208s user 0.003s sys 0.019s
I20250625 07:52:28.386843 1162 raft_consensus.cc:357] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.387022 1162 raft_consensus.cc:383] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.387063 1162 raft_consensus.cc:738] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:28.387689 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f1983df4902f48d2b408a99b440368e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1983df4902f48d2b408a99b440368e7 already in progress: creating tablet
W20250625 07:52:28.388127 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 78f0921af91140c6989de2306326c28f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78f0921af91140c6989de2306326c28f already in progress: creating tablet
W20250625 07:52:28.388438 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f5ab0400081848dd872a429685e3b8e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ab0400081848dd872a429685e3b8e2 already in progress: creating tablet
I20250625 07:52:28.388553 1166 tablet_replica.cc:331] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:28.388726 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a3e6a2f10bc14896b6bc01bdc0da3630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e6a2f10bc14896b6bc01bdc0da3630 already in progress: creating tablet
I20250625 07:52:28.388775 1166 raft_consensus.cc:2241] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.388919 1166 raft_consensus.cc:2270] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.389133 1162 consensus_queue.cc:260] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.390163 1166 ts_tablet_manager.cc:1905] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:28.395327 713 consensus_peers.cc:487] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:28.395648 1162 ts_tablet_manager.cc:1428] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 07:52:28.397186 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.397591 1162 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.401528 1162 tablet_bootstrap.cc:654] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.403271 588 master_service.cc:948] ReplaceTablet: received request to replace tablet c3d8b6b5c06248879e6052f6ae7fe46e from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:28.405304 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67ea9c5fac904443a0e6004f8bebedbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.411409 890 tablet_service.cc:1468] Processing CreateTablet for tablet 27768205b8e24a09a2b0e02153781fcf (DEFAULT_TABLE table=default.Table_d02d6575202148d0bad916a7cba0191d [id=dd78e1d773e84f50904fef5c19cf6f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.411928 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27768205b8e24a09a2b0e02153781fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.415019 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c61750d1b097487aa364f4b32693709e" } new_table_name: "default.Table_027918835ac5440cb3e80ee702033d60" modify_external_catalogs: true
I20250625 07:52:28.416429 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f63fdf2c8464c8fb58f412263d4e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:28.417874 1166 ts_tablet_manager.cc:1918] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.418018 1166 log.cc:1199] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c8c015c202544914b74f1e245b496f82
I20250625 07:52:28.418366 1166 ts_tablet_manager.cc:1939] T c8c015c202544914b74f1e245b496f82 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.419271 1162 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.419389 1162 ts_tablet_manager.cc:1397] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250625 07:52:28.420225 1166 tablet_replica.cc:331] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.420189 1162 raft_consensus.cc:357] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.420360 1162 raft_consensus.cc:383] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.420384 1166 raft_consensus.cc:2241] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.420410 1162 raft_consensus.cc:738] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.420503 1166 raft_consensus.cc:2270] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.420622 1162 consensus_queue.cc:260] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.422802 1162 ts_tablet_manager.cc:1428] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:28.423064 1162 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.423928 892 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:28.416535 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ce15eace6bc5411d808b723c797ed21e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.425529 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce15eace6bc5411d808b723c797ed21e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.426929 1162 tablet_bootstrap.cc:654] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.420971 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f6785410b52a4eb7a08b256bae5d35e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.427281 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_772868a19c60415e8d8d18220e78ab27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.427911 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_772868a19c60415e8d8d18220e78ab27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.423235 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_71d91cadf45d4ae1a05a5db26871f709"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.428471 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71d91cadf45d4ae1a05a5db26871f709 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.428573 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 866) took 1761 ms. Trace:
I20250625 07:52:28.423607 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e53a5a947f2c4463a268d7c905ee326f" } modify_external_catalogs: true
I20250625 07:52:28.428987 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ba10e2f968a543d4a5173431533d7e72" } modify_external_catalogs: true
I20250625 07:52:28.429056 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e53a5a947f2c4463a268d7c905ee326f" } modify_external_catalogs: true
I20250625 07:52:28.429189 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ba10e2f968a543d4a5173431533d7e72" } modify_external_catalogs: true
I20250625 07:52:28.423938 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cc8b946dd3564d4b95fa7174c5c9bd96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.427927 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6785410b52a4eb7a08b256bae5d35e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.429788 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc8b946dd3564d4b95fa7174c5c9bd96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.430328 1158 tablet_bootstrap.cc:654] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.424250 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0885961127594fcb9cb80fba76deceee" } modify_external_catalogs: true
I20250625 07:52:28.430944 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0885961127594fcb9cb80fba76deceee" } modify_external_catalogs: true
I20250625 07:52:28.425014 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c8c015c202544914b74f1e245b496f82 (table default.Table_2b519d271b96497dafe0b274a8755066 [id=4a285d0804ea403bb60aa49bfde4bd9a]) successfully deleted
I20250625 07:52:28.431599 1166 ts_tablet_manager.cc:1905] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:28.431740 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b7c5c675d0a7470e95a30a499ad15714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c5c675d0a7470e95a30a499ad15714 already in progress: creating tablet
I20250625 07:52:28.428654 1160 rpcz_store.cc:276] 0625 07:52:26.666928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.666989 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:28.105709 (+1438720us) ts_tablet_manager.cc:1145] Deleting tablet ba0735d576f947d395bee8d1b865f2fd
0625 07:52:28.105718 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.113040 (+ 7322us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.115833 (+ 2793us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.428543 (+312710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438524}
I20250625 07:52:28.432127 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 867) took 1688 ms. Trace:
I20250625 07:52:28.432226 1160 rpcz_store.cc:276] 0625 07:52:26.743309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.743391 (+ 82us) service_pool.cc:225] Handling call
0625 07:52:28.431960 (+1688569us) ts_tablet_manager.cc:1145] Deleting tablet 8c2ced3d2fba4f74a5d89d4bc5580905
0625 07:52:28.431974 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.432104 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1688334}
I20250625 07:52:28.432523 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 868) took 1667 ms. Trace:
I20250625 07:52:28.439738 594 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 02ec8c9e6340497abe802c7345f42921 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]). Expected version 5 got 0
I20250625 07:52:28.440611 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_54c26dc284d04c5b954ad6912d7345aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.441234 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c26dc284d04c5b954ad6912d7345aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.444135 1925 raft_consensus.cc:491] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.444299 1925 raft_consensus.cc:513] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.445230 1925 leader_election.cc:290] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.445590 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e60d4f9aed44082b9905fbb44ffb110" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:28.445887 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e60d4f9aed44082b9905fbb44ffb110" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:28.446111 851 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.446560 754 tablet_service.cc:1468] Processing CreateTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 (DEFAULT_TABLE table=default.Table_3613c8cb799746c294a74ca856714124 [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.446863 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889 (DEFAULT_TABLE table=default.Table_cb1093429f4646c7ade95e18696a76bc [id=dada44a021dd4b2a900f34153e211d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.446969 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba8780a44864169ac9cd63ab2e89fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.447192 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.447474 740 tablet_service.cc:1468] Processing CreateTablet for tablet ec2459d73e594a0c9622acc14bf1c5b5 (DEFAULT_TABLE table=default.Table_f39da4a74c09429db8be207816d8f5d1 [id=5a1b45d98e574363a7cc95a244a93d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.447780 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2459d73e594a0c9622acc14bf1c5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.448113 755 tablet_service.cc:1468] Processing CreateTablet for tablet 81cd2b2215cd4a17add2306d01d99b9a (DEFAULT_TABLE table=default.Table_5a68ddb21adc44d584bf839cb84d1c18 [id=687ef996cfaf4f1787d709d3ff0fcccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.448433 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cd2b2215cd4a17add2306d01d99b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.450666 748 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 (DEFAULT_TABLE table=default.Table_1a052ae2f0774e85bdc42172e61770d1 [id=1bad4ceb6f1642da98e2aa014e78a1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.450969 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3eafac1b2b46acb2ac6fca087b7f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.451496 746 tablet_service.cc:1468] Processing CreateTablet for tablet 448a7d67fc014411ab294b19c65637ee (DEFAULT_TABLE table=default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8 [id=38cb713de33f4852a9930e19c378c20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.451787 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448a7d67fc014411ab294b19c65637ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.452916 752 tablet_service.cc:1468] Processing CreateTablet for tablet 1488eddcbbc84562a2bcb73099161de8 (DEFAULT_TABLE table=default.Table_2166d8d523c34441943a1d577b745b4d [id=452b083a9563462ba0bcf108fcec8ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.452947 758 tablet_service.cc:1468] Processing CreateTablet for tablet e34e54db2181402dab9b6c5b22bc7da9 (DEFAULT_TABLE table=default.Table_fa33440d4d5740c38f6730f962447c3e [id=b07e7cda7a814f57a6aa51edc0d049c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.453210 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1488eddcbbc84562a2bcb73099161de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.453256 750 tablet_service.cc:1468] Processing CreateTablet for tablet 07f0161046bd42cbb3ecbcf004c13891 (DEFAULT_TABLE table=default.Table_b2e47db078dd4a8c8ec4c8df33955362 [id=cb56a7d28b144e2884b80c21be1b3416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.453310 1733 raft_consensus.cc:491] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.453536 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f0161046bd42cbb3ecbcf004c13891. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.453438 1733 raft_consensus.cc:513] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.454067 1733 leader_election.cc:290] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:28.454888 986 leader_election.cc:343] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a777f64425384be5bccc86b197626618
I20250625 07:52:28.455087 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34e54db2181402dab9b6c5b22bc7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.452947 739 tablet_service.cc:1468] Processing CreateTablet for tablet 6ab9a28007e74d9cbd1c7affbf5c660c (DEFAULT_TABLE table=default.Table_dd10ce6555604cb084e1b1d299cf9aa2 [id=d30ee521df404093a399d5bf600cf3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.456936 756 tablet_service.cc:1468] Processing CreateTablet for tablet de3b07b2e0284761a96ea3650506638b (DEFAULT_TABLE table=default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a [id=aae15985ebae4792b7f74e6f31e8ac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.457043 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab9a28007e74d9cbd1c7affbf5c660c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.457226 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3b07b2e0284761a96ea3650506638b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.458813 746 tablet_service.cc:1468] Processing CreateTablet for tablet f406d1c8777f474c97f6e804cb230bb6 (DEFAULT_TABLE table=default.Table_006e977b28ff4dbe9f2091ba6f1879e2 [id=0621b19752f041429ae5533ce4f57566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.459201 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406d1c8777f474c97f6e804cb230bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.460927 751 tablet_service.cc:1468] Processing CreateTablet for tablet 5986bb52335c46bf845c48cb044495b1 (DEFAULT_TABLE table=default.Table_5ddc30bfdeea430e8109fea6979694a5 [id=2307d739c95a4d0ca4efe1d533f65545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.461248 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5986bb52335c46bf845c48cb044495b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.461786 850 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.461905 850 leader_election.cc:304] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.462162 1925 raft_consensus.cc:2747] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.463719 758 tablet_service.cc:1468] Processing CreateTablet for tablet cd0b0642010a43f5a81748b7682f6a67 (DEFAULT_TABLE table=default.Table_98f0756428904e40a7406141b5194756 [id=dd5b69cfe373405d9b1acff25a920e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.464020 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0b0642010a43f5a81748b7682f6a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.464462 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 (DEFAULT_TABLE table=default.Table_4e2dc27726d944c5885d62ed4258acfc [id=0779497b504d4295938af3db616b1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.464546 749 tablet_service.cc:1468] Processing CreateTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e (DEFAULT_TABLE table=default.Table_602ca03b4f824a7686588dc776b8bed6 [id=91d401787d5d4f09948941ecc4b1a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.464782 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d34bfe377ba421daede5ae68848a2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.464932 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113c05d81d2943e38bcd62bf83a46a2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.465415 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b927ec564efa4ff080d22a673abb9b57 consensus state has no leader
I20250625 07:52:28.466610 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 02ec8c9e6340497abe802c7345f42921 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]). Expected version 5 got 0
I20250625 07:52:28.467581 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b2ab9e4f189492dbfec4ec5c79a1630" } modify_external_catalogs: true
I20250625 07:52:28.467808 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2b2ab9e4f189492dbfec4ec5c79a1630" } modify_external_catalogs: true
I20250625 07:52:28.432585 1160 rpcz_store.cc:276] 0625 07:52:26.766617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.766968 (+ 351us) service_pool.cc:225] Handling call
0625 07:52:28.432422 (+1665454us) ts_tablet_manager.cc:1145] Deleting tablet d605113ad8f246148825923e5bcdf00d
0625 07:52:28.432430 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.432509 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665262}
W20250625 07:52:28.468848 985 leader_election.cc:343] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a777f64425384be5bccc86b197626618
I20250625 07:52:28.468927 1160 tablet_replica.cc:331] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.468972 985 leader_election.cc:304] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:28.469223 1733 raft_consensus.cc:2747] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.469121 1160 raft_consensus.cc:2241] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.469516 747 tablet_service.cc:1468] Processing CreateTablet for tablet f8327a1aeb2743b3993ffac98b6ca2c0 (DEFAULT_TABLE table=default.Table_ca8eeef7078a4df39792b710bafcbd22 [id=a1c720dea2684e9c8e5192084e88686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.469686 1160 raft_consensus.cc:2270] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.469831 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8327a1aeb2743b3993ffac98b6ca2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.470425 1160 ts_tablet_manager.cc:1905] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.474187 753 tablet_service.cc:1468] Processing CreateTablet for tablet b45add1fbcc2423d92b7b0001f946cb6 (DEFAULT_TABLE table=default.Table_e2e1925e33414e199b380b98a0c6e96f [id=6cd6d9a3c91a44e7a8d454777677d909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.474541 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45add1fbcc2423d92b7b0001f946cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.481931 753 tablet_service.cc:1468] Processing CreateTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 (DEFAULT_TABLE table=default.Table_8319e385441549e98fc4f6f600dd17b7 [id=5a3e037b6b7944968b285f678c967d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.482302 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b1ec100ca4488c9d7fc470246ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.482820 1158 tablet_bootstrap.cc:492] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:28.482964 1158 ts_tablet_manager.cc:1397] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.175s user 0.004s sys 0.000s
I20250625 07:52:28.483724 1158 raft_consensus.cc:357] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.483901 1158 raft_consensus.cc:383] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.483974 1158 raft_consensus.cc:738] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.484223 1158 consensus_queue.cc:260] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.487046 1162 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.487265 1162 ts_tablet_manager.cc:1397] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.000s
I20250625 07:52:28.488235 1162 raft_consensus.cc:357] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.488505 1162 raft_consensus.cc:383] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.488606 1162 raft_consensus.cc:738] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.488905 1162 consensus_queue.cc:260] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.489262 1166 ts_tablet_manager.cc:1918] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.489399 1166 log.cc:1199] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:28.489713 1166 ts_tablet_manager.cc:1939] T 3ca0144e3c5142ac9fcbd927b7585a3c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.491852 1162 ts_tablet_manager.cc:1428] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 07:52:28.492247 1162 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.494122 1162 tablet_bootstrap.cc:654] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.499167 749 tablet_service.cc:1468] Processing CreateTablet for tablet 190c4bfe74344994babdc7e94513e6c2 (DEFAULT_TABLE table=default.Table_90f97f84512542b29acecb9db3e1172d [id=0fd4ed98e556473292ed565b2a60ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.499621 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190c4bfe74344994babdc7e94513e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.499732 1161 ts_tablet_manager.cc:1918] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.499848 1161 log.cc:1199] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/5c28a064006145eea000e74da10d0794
I20250625 07:52:28.500144 1161 ts_tablet_manager.cc:1939] T 5c28a064006145eea000e74da10d0794 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:28.501013 1161 tablet_replica.cc:331] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:28.501207 1161 raft_consensus.cc:2241] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.501365 1161 raft_consensus.cc:2270] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.502053 1161 ts_tablet_manager.cc:1905] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.502328 755 tablet_service.cc:1468] Processing CreateTablet for tablet 5755b609b94f446f83341c16870ac1c1 (DEFAULT_TABLE table=default.Table_6aceb5dc89a54074b4b33990008cb85e [id=621ed69e94d24dc28714b0ce7ae93987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.502707 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5755b609b94f446f83341c16870ac1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.507763 741 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c (DEFAULT_TABLE table=default.Table_895827fb99994402b75be290df98a2ff [id=e8069108ae2b4dd392d7253ae8e6d4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.508378 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.509999 1158 ts_tablet_manager.cc:1428] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250625 07:52:28.510308 1158 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:28.511099 1166 tablet_replica.cc:331] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.511304 1166 raft_consensus.cc:2241] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.511483 1166 raft_consensus.cc:2270] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.511634 1158 tablet_bootstrap.cc:654] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.517827 757 tablet_service.cc:1468] Processing CreateTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 (DEFAULT_TABLE table=default.Table_c3197b8d0402490b86e0fbc10c91c62e [id=aee83d7d1b9a4ac7aac0d19123403db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.518224 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fa6e5ac74444729645a27fc2945bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.518164 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ed4947df987b4ce48a2209dea3bf6dee" } modify_external_catalogs: true
I20250625 07:52:28.518302 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3ca0144e3c5142ac9fcbd927b7585a3c (table default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]) successfully deleted
I20250625 07:52:28.518358 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ed4947df987b4ce48a2209dea3bf6dee" } modify_external_catalogs: true
I20250625 07:52:28.519798 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_66a34579b0ea4afca2c08484ccee0b1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.520382 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66a34579b0ea4afca2c08484ccee0b1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.523562 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fdad7dc1b27e4d87a747642a6d47230c
I20250625 07:52:28.525017 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8d4a225cda044dba96006acca0260845" } new_table_name: "default.Table_6444b00ef40d4993b4e6cb5dca41f983" modify_external_catalogs: true
I20250625 07:52:28.525092 754 tablet_service.cc:1468] Processing CreateTablet for tablet e609e7267a66453dab2b666d69aebd02 (DEFAULT_TABLE table=default.Table_f62f60f959ba405b8f9f91731541cee3 [id=0c11b58976464c94967abf87773d89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.525478 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e609e7267a66453dab2b666d69aebd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.526583 1733 raft_consensus.cc:491] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.526611 741 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.526769 1733 raft_consensus.cc:513] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.527428 1733 leader_election.cc:290] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:28.527462 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 51744ae4b84e47cea49fb54023777a41
I20250625 07:52:28.527572 1733 raft_consensus.cc:491] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.527685 1733 raft_consensus.cc:513] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.528198 1733 leader_election.cc:290] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:28.528672 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1a653e84823d467f9beb897f2bfb3b3b
W20250625 07:52:28.529531 985 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2f63fdf2c8464c8fb58f412263d4e5c5
W20250625 07:52:28.529734 985 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:28.530251 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.530548 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 51744ae4b84e47cea49fb54023777a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.531965 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0146bda63784b9b8882071ad0c5c931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.532595 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0146bda63784b9b8882071ad0c5c931 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.532891 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ed4bfc88fee44f85afe168b7dcd1017e" } new_table_name: "default.Table_f7ce218193d24d2b96f16895ea6ac04a" modify_external_catalogs: true
I20250625 07:52:28.533248 1166 ts_tablet_manager.cc:1905] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:28.533290 986 leader_election.cc:343] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:28.533401 986 leader_election.cc:304] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:28.533772 986 leader_election.cc:343] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2f63fdf2c8464c8fb58f412263d4e5c5
I20250625 07:52:28.533878 986 leader_election.cc:304] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:28.534128 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b80453878cf143caba5ec40c14087313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.534241 1733 raft_consensus.cc:2747] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.534463 1976 raft_consensus.cc:2747] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:28.534729 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b80453878cf143caba5ec40c14087313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.537014 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d86f89948c7043899036f6af6183d950" } new_table_name: "default.Table_9c15ba9c8f684ea4bc03053e7370eefe" modify_external_catalogs: true
I20250625 07:52:28.538220 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_380bf0b7bfc248f7b340c14b230c4db7" } new_table_name: "default.Table_21de3cd434744102b0d5ffbbc17c18c3" modify_external_catalogs: true
I20250625 07:52:28.538973 1166 ts_tablet_manager.cc:1918] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.539089 1166 log.cc:1199] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/450727d8f33e4eb489969b7974cdeeb1
I20250625 07:52:28.539520 1166 ts_tablet_manager.cc:1939] T 450727d8f33e4eb489969b7974cdeeb1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.540470 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 450727d8f33e4eb489969b7974cdeeb1 (table default.Table_5c23014bc5574cc9a64e940e87188050 [id=01cb78211cdc4a9a9b493e31f8c44a99]) successfully deleted
I20250625 07:52:28.541126 1166 tablet_replica.cc:331] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.541193 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4243f6f8c1154fc89b3aa1874441b98d
I20250625 07:52:28.543947 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.545320 1160 ts_tablet_manager.cc:1918] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.546562 1160 log.cc:1199] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c8c015c202544914b74f1e245b496f82
I20250625 07:52:28.546878 1160 ts_tablet_manager.cc:1939] T c8c015c202544914b74f1e245b496f82 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:28.546914 713 consensus_peers.cc:487] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:28.547816 1162 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.547963 1162 ts_tablet_manager.cc:1397] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20250625 07:52:28.549016 1162 raft_consensus.cc:357] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.549192 1162 raft_consensus.cc:383] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.549247 1162 raft_consensus.cc:738] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.549445 1162 consensus_queue.cc:260] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.550433 1162 ts_tablet_manager.cc:1428] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 07:52:28.551784 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94abb9c8a70a41169fb602c159d92f68 consensus state has no leader
I20250625 07:52:28.552999 753 tablet_service.cc:1468] Processing CreateTablet for tablet bff347d838fe4e56a195a957d0deb742 (DEFAULT_TABLE table=default.Table_2f8acc41371a4800a82de306adce74db [id=b6406562b9b14056ba9b938c38e2fe89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.553238 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_34b97f57ddc44127b6e68fb8a5542a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.553900 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34b97f57ddc44127b6e68fb8a5542a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.556840 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0cdee1535e5a4a21804e28e78aaea0eb (table default.Table_4a4b98165527418ea3ccc846b86e4d15 [id=fb8226e9c0a440629bae51d5425f6f31], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0cdee1535e5a4a21804e28e78aaea0eb consensus state has no leader
I20250625 07:52:28.557766 755 tablet_service.cc:1468] Processing CreateTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 (DEFAULT_TABLE table=default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.559814 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 381ae5ef5df9437d9bfe0a51987dfe62
I20250625 07:52:28.560457 754 tablet_service.cc:1468] Processing CreateTablet for tablet b85adedf7d2c45a1b33ee4a5633ee734 (DEFAULT_TABLE table=default.Table_3b5acb6fb6cd40d2855892e2d28bed37 [id=d87af998b5ff47d69efb9780b11feaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.560573 757 tablet_service.cc:1468] Processing CreateTablet for tablet 65caf04d39a044ec82514f309128e767 (DEFAULT_TABLE table=default.Table_09bebf0eae58464c970e326392341524 [id=d90831797f054efcb9b5526e6d3c2cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.560773 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.563604 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 869) took 1780 ms. Trace:
I20250625 07:52:28.563712 1160 rpcz_store.cc:276] 0625 07:52:26.783168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.783253 (+ 85us) service_pool.cc:225] Handling call
0625 07:52:28.468875 (+1685622us) ts_tablet_manager.cc:1145] Deleting tablet c8c015c202544914b74f1e245b496f82
0625 07:52:28.468885 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.513927 (+ 45042us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.545201 (+ 31274us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.563579 (+ 18378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1685418}
I20250625 07:52:28.564181 1160 tablet_replica.cc:331] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.564371 1160 raft_consensus.cc:2241] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.564527 1160 raft_consensus.cc:2270] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.565025 1160 ts_tablet_manager.cc:1905] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.566008 1925 raft_consensus.cc:491] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.566136 1925 raft_consensus.cc:513] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.566736 1925 leader_election.cc:290] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.567184 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:28.567265 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.567384 777 raft_consensus.cc:2466] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
W20250625 07:52:28.567693 850 leader_election.cc:343] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.567811 850 leader_election.cc:304] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.568217 1925 raft_consensus.cc:2802] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:28.568295 1925 raft_consensus.cc:491] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:28.568354 1925 raft_consensus.cc:3058] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:28.570750 1925 raft_consensus.cc:513] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.571759 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:28.571938 777 raft_consensus.cc:3058] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:28.574479 777 raft_consensus.cc:2466] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:28.575085 851 leader_election.cc:304] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:28.575348 1793 raft_consensus.cc:2802] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:28.576927 1925 leader_election.cc:290] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.577047 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8fe6405c0ea4d6ca0257a5b485251fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:28.577474 850 leader_election.cc:343] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.588506 1158 tablet_bootstrap.cc:492] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:28.588524 1162 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.588943 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff347d838fe4e56a195a957d0deb742. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.589563 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8f25febb41c403ca8be268206157756 consensus state has no leader
I20250625 07:52:28.590188 1158 ts_tablet_manager.cc:1397] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20250625 07:52:28.591228 1158 raft_consensus.cc:357] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.591418 1158 raft_consensus.cc:383] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.591485 1158 raft_consensus.cc:738] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.591712 1158 consensus_queue.cc:260] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.592200 1158 ts_tablet_manager.cc:1428] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:28.592912 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dff56bc7445abcd6b7c1cb525553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.595163 1163 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:28.595278 1163 ts_tablet_manager.cc:1397] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.493s user 0.004s sys 0.016s
I20250625 07:52:28.596967 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85adedf7d2c45a1b33ee4a5633ee734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.595979 1163 raft_consensus.cc:357] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.598763 1163 raft_consensus.cc:383] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.598829 1163 raft_consensus.cc:738] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.599028 1163 consensus_queue.cc:260] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.601004 1163 ts_tablet_manager.cc:1428] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:28.600970 1856 raft_consensus.cc:491] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.601120 1856 raft_consensus.cc:513] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.601290 1163 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:28.601819 1856 leader_election.cc:290] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.602990 1161 ts_tablet_manager.cc:1918] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.603120 1161 log.cc:1199] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/edda0c3f60f64ed0b965eca52c5b3591
I20250625 07:52:28.603457 1161 ts_tablet_manager.cc:1939] T edda0c3f60f64ed0b965eca52c5b3591 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:28.606868 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65caf04d39a044ec82514f309128e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.608040 1163 tablet_bootstrap.cc:654] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.615064 1160 ts_tablet_manager.cc:1918] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.615190 1160 log.cc:1199] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:28.615448 1160 ts_tablet_manager.cc:1939] T 5ec29e4294864fbabdff5a04dcf5b163 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.616535 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 870) took 1781 ms. Trace:
I20250625 07:52:28.616624 1160 rpcz_store.cc:276] 0625 07:52:26.835476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.835681 (+ 205us) service_pool.cc:225] Handling call
0625 07:52:28.564125 (+1728444us) ts_tablet_manager.cc:1145] Deleting tablet 5ec29e4294864fbabdff5a04dcf5b163
0625 07:52:28.564146 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.606034 (+ 41888us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.614963 (+ 8929us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.616511 (+ 1548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728227}
I20250625 07:52:28.617148 1160 tablet_replica.cc:331] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.617331 1160 raft_consensus.cc:2241] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.617470 1160 raft_consensus.cc:2270] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.617460 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet ba0735d576f947d395bee8d1b865f2fd (table default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]) successfully deleted
W20250625 07:52:28.617679 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8c2ced3d2fba4f74a5d89d4bc5580905 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c2ced3d2fba4f74a5d89d4bc5580905 already in progress: creating tablet
W20250625 07:52:28.617868 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d605113ad8f246148825923e5bcdf00d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d605113ad8f246148825923e5bcdf00d already in progress: creating tablet
I20250625 07:52:28.617933 1160 ts_tablet_manager.cc:1905] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.618158 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c8c015c202544914b74f1e245b496f82 (table default.Table_2b519d271b96497dafe0b274a8755066 [id=4a285d0804ea403bb60aa49bfde4bd9a]) successfully deleted
I20250625 07:52:28.618497 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5ec29e4294864fbabdff5a04dcf5b163 (table default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]) successfully deleted
I20250625 07:52:28.619201 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889 (DEFAULT_TABLE table=default.Table_cb1093429f4646c7ade95e18696a76bc [id=dada44a021dd4b2a900f34153e211d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.619740 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a33b7cab19244ec08d3821b6cc4c072d" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.620492 1163 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:28.620985 1163 ts_tablet_manager.cc:1397] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
W20250625 07:52:28.621057 713 leader_election.cc:343] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:28.621300 713 leader_election.cc:343] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:28.621394 713 leader_election.cc:304] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.621963 1163 raft_consensus.cc:357] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.622583 1163 raft_consensus.cc:383] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.622875 1163 raft_consensus.cc:738] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.623351 1163 consensus_queue.cc:260] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.624306 1163 ts_tablet_manager.cc:1428] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250625 07:52:28.624745 1160 ts_tablet_manager.cc:1918] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.624874 1160 log.cc:1199] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/be7b755d95b14ecea7a691003455f381
I20250625 07:52:28.625161 1163 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:28.625983 1982 raft_consensus.cc:2747] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.625217 1160 ts_tablet_manager.cc:1939] T be7b755d95b14ecea7a691003455f381 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.626214 1793 raft_consensus.cc:695] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:28.626525 1793 consensus_queue.cc:237] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.626834 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 871) took 1768 ms. Trace:
I20250625 07:52:28.626911 1162 tablet_bootstrap.cc:654] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.626935 1160 rpcz_store.cc:276] 0625 07:52:26.858318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.858412 (+ 94us) service_pool.cc:225] Handling call
0625 07:52:28.617101 (+1758689us) ts_tablet_manager.cc:1145] Deleting tablet be7b755d95b14ecea7a691003455f381
0625 07:52:28.617111 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.621333 (+ 4222us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.624630 (+ 3297us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.626813 (+ 2183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758493}
I20250625 07:52:28.627422 1160 tablet_replica.cc:331] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.627637 1160 raft_consensus.cc:2241] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.627810 1160 raft_consensus.cc:2270] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.628402 1160 ts_tablet_manager.cc:1905] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.629582 1982 consensus_queue.cc:1035] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:28.628559 877 tablet_service.cc:1468] Processing CreateTablet for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889 (DEFAULT_TABLE table=default.Table_cb1093429f4646c7ade95e18696a76bc [id=dada44a021dd4b2a900f34153e211d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.632056 1970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38477 every 8 connection(s)
I20250625 07:52:28.632508 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.636106 1160 ts_tablet_manager.cc:1918] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.636224 1160 log.cc:1199] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1898ff5b7626478bb997f7a433477498
I20250625 07:52:28.636502 1160 ts_tablet_manager.cc:1939] T 1898ff5b7626478bb997f7a433477498 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.637533 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 872) took 1708 ms. Trace:
I20250625 07:52:28.637650 1160 rpcz_store.cc:276] 0625 07:52:26.929524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.929576 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:28.627377 (+1697801us) ts_tablet_manager.cc:1145] Deleting tablet 1898ff5b7626478bb997f7a433477498
0625 07:52:28.627386 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.632845 (+ 5459us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.636016 (+ 3171us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.637507 (+ 1491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697585}
I20250625 07:52:28.638222 1160 tablet_replica.cc:331] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.638458 1160 raft_consensus.cc:2241] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.638624 1160 raft_consensus.cc:2270] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.639240 1160 ts_tablet_manager.cc:1905] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.645763 1160 ts_tablet_manager.cc:1918] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.645891 1160 log.cc:1199] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/f191436f3b9244eaa12d211b6f125175
I20250625 07:52:28.646204 1160 ts_tablet_manager.cc:1939] T f191436f3b9244eaa12d211b6f125175 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.646922 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 873) took 1675 ms. Trace:
I20250625 07:52:28.647020 1160 rpcz_store.cc:276] 0625 07:52:26.971958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:26.972022 (+ 64us) service_pool.cc:225] Handling call
0625 07:52:28.638151 (+1666129us) ts_tablet_manager.cc:1145] Deleting tablet f191436f3b9244eaa12d211b6f125175
0625 07:52:28.638164 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.642525 (+ 4361us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.645678 (+ 3153us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.646902 (+ 1224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665920}
I20250625 07:52:28.647562 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 886) took 1445 ms. Trace:
I20250625 07:52:28.647648 1160 rpcz_store.cc:276] 0625 07:52:27.202252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.203115 (+ 863us) service_pool.cc:225] Handling call
0625 07:52:28.647437 (+1444322us) ts_tablet_manager.cc:1145] Deleting tablet 4167282133f94bfb9870bd08fc6594b4
0625 07:52:28.647447 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.647545 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444149}
I20250625 07:52:28.648243 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 887) took 1444 ms. Trace:
I20250625 07:52:28.648325 1160 rpcz_store.cc:276] 0625 07:52:27.203677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.203733 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:28.648131 (+1444398us) ts_tablet_manager.cc:1145] Deleting tablet f04016cda7f746d49e30be3b1c9c18b6
0625 07:52:28.648140 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.648228 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444255}
I20250625 07:52:28.648840 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 888) took 1442 ms. Trace:
I20250625 07:52:28.648907 1160 rpcz_store.cc:276] 0625 07:52:27.205903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.205956 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:28.648730 (+1442774us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d357cefb84a239d5be73857b9097b
0625 07:52:28.648742 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.648828 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442556}
I20250625 07:52:28.649408 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 889) took 1440 ms. Trace:
I20250625 07:52:28.649469 1160 rpcz_store.cc:276] 0625 07:52:27.209321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.209527 (+ 206us) service_pool.cc:225] Handling call
0625 07:52:28.649321 (+1439794us) ts_tablet_manager.cc:1145] Deleting tablet 52c1f7cdd88d423788d7d30f15318786
0625 07:52:28.649329 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.649395 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439634}
I20250625 07:52:28.649578 1023 tablet_service.cc:1468] Processing CreateTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 (DEFAULT_TABLE table=default.Table_3613c8cb799746c294a74ca856714124 [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.649833 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 900) took 1267 ms. Trace:
I20250625 07:52:28.649896 1160 rpcz_store.cc:276] 0625 07:52:27.382783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.383125 (+ 342us) service_pool.cc:225] Handling call
0625 07:52:28.649745 (+1266620us) ts_tablet_manager.cc:1145] Deleting tablet ccd929ffd026433c9abd3851bb8e1004
0625 07:52:28.649754 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.649821 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266445}
I20250625 07:52:28.650214 1160 tablet_replica.cc:331] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.650395 1160 raft_consensus.cc:2241] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.650560 1160 raft_consensus.cc:2270] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.651065 1160 ts_tablet_manager.cc:1905] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.652784 877 tablet_service.cc:1468] Processing CreateTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 (DEFAULT_TABLE table=default.Table_3613c8cb799746c294a74ca856714124 [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.655011 1158 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:28.656641 1158 tablet_bootstrap.cc:654] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.658922 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 873) took 1075 ms. Trace:
I20250625 07:52:28.659216 1161 rpcz_store.cc:276] 0625 07:52:27.583262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.583305 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:28.500970 (+917665us) ts_tablet_manager.cc:1145] Deleting tablet edda0c3f60f64ed0b965eca52c5b3591
0625 07:52:28.500978 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.554034 (+ 53056us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.602855 (+ 48821us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.658848 (+ 55993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":917526}
I20250625 07:52:28.660908 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 874) took 1077 ms. Trace:
I20250625 07:52:28.661167 1161 rpcz_store.cc:276] 0625 07:52:27.583678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.583722 (+ 44us) service_pool.cc:225] Handling call
0625 07:52:28.659948 (+1076226us) ts_tablet_manager.cc:1145] Deleting tablet a396bde4b5964697ad0090eab777ba0e
0625 07:52:28.660645 (+ 697us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.660842 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076104}
I20250625 07:52:28.662382 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 875) took 1078 ms. Trace:
I20250625 07:52:28.662623 1161 rpcz_store.cc:276] 0625 07:52:27.584316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.584358 (+ 42us) service_pool.cc:225] Handling call
0625 07:52:28.661839 (+1077481us) ts_tablet_manager.cc:1145] Deleting tablet 2f5fec0e5f364ca093879a8ad2e0bcfc
0625 07:52:28.662164 (+ 325us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.662317 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1077354}
I20250625 07:52:28.663645 1161 tablet_replica.cc:331] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:28.665232 1687 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38477
I20250625 07:52:28.665210 1161 raft_consensus.cc:2241] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:28.667364 713 consensus_peers.cc:487] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:28.666872 1161 raft_consensus.cc:2270] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.668263 1687 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-0/data/info.pb
W20250625 07:52:28.669839 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 768c9551d33346718af2670a01a626b0 consensus state has no leader
I20250625 07:52:28.670776 1161 ts_tablet_manager.cc:1905] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.671967 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_adb5d0acb48c45b781c468e36e59b4c0" } new_table_name: "default.Table_6fbc60d9bf1c4a169d7565b1d870e351" modify_external_catalogs: true
I20250625 07:52:28.673770 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu as pid 1687
I20250625 07:52:28.674108 1166 raft_consensus.cc:2241] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.674973 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 3ab38e0cdcd14f209e6877f2e4a46e98 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:28.675082 1166 raft_consensus.cc:2270] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.679009 1166 ts_tablet_manager.cc:1905] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.683833 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8dcfa4fe6045f0a29fc2a61cd4a889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.688441 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba8780a44864169ac9cd63ab2e89fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.691711 1981 consensus_queue.cc:1035] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:28.703678 1161 ts_tablet_manager.cc:1918] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:28.703687 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08cd482ceec244c2bed2b116d5c14d89 consensus state has no leader
I20250625 07:52:28.657725 1160 ts_tablet_manager.cc:1918] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.705902 895 tablet_service.cc:1468] Processing CreateTablet for tablet ec2459d73e594a0c9622acc14bf1c5b5 (DEFAULT_TABLE table=default.Table_f39da4a74c09429db8be207816d8f5d1 [id=5a1b45d98e574363a7cc95a244a93d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.707711 890 tablet_service.cc:1468] Processing CreateTablet for tablet 1488eddcbbc84562a2bcb73099161de8 (DEFAULT_TABLE table=default.Table_2166d8d523c34441943a1d577b745b4d [id=452b083a9563462ba0bcf108fcec8ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.709461 1160 log.cc:1199] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7480c07af7ce4a3889bd8d42d7419d57
I20250625 07:52:28.709702 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1488eddcbbc84562a2bcb73099161de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.709821 1160 ts_tablet_manager.cc:1939] T 7480c07af7ce4a3889bd8d42d7419d57 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.712283 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4709f33f040149c49bd273524c9fd4f0 (DEFAULT_TABLE table=default.Table_71d91cadf45d4ae1a05a5db26871f709 [id=9bc20b9bd6a14d85bba8973cf718df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.712451 753 tablet_service.cc:1468] Processing CreateTablet for tablet 60edc8fa184542a0b54b89736831730d (DEFAULT_TABLE table=default.Table_cc8b946dd3564d4b95fa7174c5c9bd96 [id=f28fa7e6c39c478c9b313cb0b0963fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.712662 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4709f33f040149c49bd273524c9fd4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.712893 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60edc8fa184542a0b54b89736831730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.715845 1162 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.715983 1162 ts_tablet_manager.cc:1397] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.128s user 0.006s sys 0.000s
I20250625 07:52:28.718585 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 907) took 1210 ms. Trace:
I20250625 07:52:28.718680 1160 rpcz_store.cc:276] 0625 07:52:27.508591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.516305 (+ 7714us) service_pool.cc:225] Handling call
0625 07:52:28.650164 (+1133859us) ts_tablet_manager.cc:1145] Deleting tablet 7480c07af7ce4a3889bd8d42d7419d57
0625 07:52:28.650172 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.654596 (+ 4424us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.657639 (+ 3043us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.718558 (+ 60919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094219}
I20250625 07:52:28.719067 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 914) took 1135 ms. Trace:
I20250625 07:52:28.719130 1160 rpcz_store.cc:276] 0625 07:52:27.584066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.584120 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:28.718946 (+1134826us) ts_tablet_manager.cc:1145] Deleting tablet a396bde4b5964697ad0090eab777ba0e
0625 07:52:28.718958 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.719053 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134671}
I20250625 07:52:28.719390 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 915) took 1134 ms. Trace:
I20250625 07:52:28.719444 1160 rpcz_store.cc:276] 0625 07:52:27.584720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.584767 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:28.719313 (+1134546us) ts_tablet_manager.cc:1145] Deleting tablet 2f5fec0e5f364ca093879a8ad2e0bcfc
0625 07:52:28.719321 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.719379 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134393}
I20250625 07:52:28.719640 1160 tablet_replica.cc:331] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.719803 1160 raft_consensus.cc:2241] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.719959 1160 raft_consensus.cc:2270] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.720461 1160 ts_tablet_manager.cc:1905] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.720461 749 tablet_service.cc:1468] Processing CreateTablet for tablet f2149b6b3e9743b9a392171626d5b4bb (DEFAULT_TABLE table=default.Table_66a34579b0ea4afca2c08484ccee0b1f [id=2028b2ce2ce0455b8e536b5c2140f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.720582 741 tablet_service.cc:1468] Processing CreateTablet for tablet 95545f3569ad49828ae49136f122e690 (DEFAULT_TABLE table=default.Table_f6785410b52a4eb7a08b256bae5d35e0 [id=c51fd8476a0e43ff86fa68acf94a86fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.720863 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2149b6b3e9743b9a392171626d5b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.721024 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95545f3569ad49828ae49136f122e690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.721483 754 tablet_service.cc:1468] Processing CreateTablet for tablet c84eab61f1254a67a528c314cd9ab899 (DEFAULT_TABLE table=default.Table_772868a19c60415e8d8d18220e78ab27 [id=7dbeece354644dc896f3c748a5bedd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.721572 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2459d73e594a0c9622acc14bf1c5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.721859 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84eab61f1254a67a528c314cd9ab899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.724956 757 tablet_service.cc:1468] Processing CreateTablet for tablet 652e07d46af74945bf504f7d9015a446 (DEFAULT_TABLE table=default.Table_00fc5f49db204824b892b2ae69796dca [id=5ff5c3c148ad47fbb1121e7c02605e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.725302 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652e07d46af74945bf504f7d9015a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.729115 755 tablet_service.cc:1468] Processing CreateTablet for tablet 494451c0c2514b08ae6be20391a7bcf8 (DEFAULT_TABLE table=default.Table_3245e577c1b349f0b11133c6d1363526 [id=a4f3a18fd7604ad882dea15863c31b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.729482 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494451c0c2514b08ae6be20391a7bcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.730706 1160 ts_tablet_manager.cc:1918] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:28.730814 1160 log.cc:1199] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/edda0c3f60f64ed0b965eca52c5b3591
I20250625 07:52:28.731130 1160 ts_tablet_manager.cc:1939] T edda0c3f60f64ed0b965eca52c5b3591 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:28.731833 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 913) took 1148 ms. Trace:
I20250625 07:52:28.731930 1160 rpcz_store.cc:276] 0625 07:52:27.583022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.584785 (+ 1763us) service_pool.cc:225] Handling call
0625 07:52:28.719606 (+1134821us) ts_tablet_manager.cc:1145] Deleting tablet edda0c3f60f64ed0b965eca52c5b3591
0625 07:52:28.719613 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.727352 (+ 7739us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.730626 (+ 3274us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.731813 (+ 1187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134622}
I20250625 07:52:28.733508 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_40a9b063009d432680f575cbc3dfad3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.734050 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a2517172003a44c492a3e3b9260bc0d5" } modify_external_catalogs: true
W20250625 07:52:28.734174 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40a9b063009d432680f575cbc3dfad3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.734289 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a2517172003a44c492a3e3b9260bc0d5" } modify_external_catalogs: true
I20250625 07:52:28.735064 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4dc958f1088e4e719b8afc3b90c3b8dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.735558 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc958f1088e4e719b8afc3b90c3b8dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.735643 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a2c86b9769ce4cb695819e5d7601ef14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.736094 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2c86b9769ce4cb695819e5d7601ef14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.736665 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7b26265e9f9a44bcbd5ba18263f61864" } modify_external_catalogs: true
I20250625 07:52:28.736851 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7b26265e9f9a44bcbd5ba18263f61864" } modify_external_catalogs: true
I20250625 07:52:28.737257 1977 raft_consensus.cc:491] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.737383 1977 raft_consensus.cc:513] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.737998 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_066619da90c542d0a712b3e36629063b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.738176 1977 leader_election.cc:290] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.738389 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f4359fd842a4760bae75169e8914f1c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:28.738451 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066619da90c542d0a712b3e36629063b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:28.738958 851 leader_election.cc:343] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.739408 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7e8eb7e9baf5497ca408e7592b57e4dc" } modify_external_catalogs: true
I20250625 07:52:28.739564 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7e8eb7e9baf5497ca408e7592b57e4dc" } modify_external_catalogs: true
I20250625 07:52:28.739497 1162 raft_consensus.cc:357] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.739652 1162 raft_consensus.cc:383] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.739706 1162 raft_consensus.cc:738] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.739917 1162 consensus_queue.cc:260] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.740721 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_23c6bf383db34b55ac2b09f3327eced1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.741261 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23c6bf383db34b55ac2b09f3327eced1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.742014 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dba8780a44864169ac9cd63ab2e89fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.745992 890 tablet_service.cc:1468] Processing CreateTablet for tablet de3b07b2e0284761a96ea3650506638b (DEFAULT_TABLE table=default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a [id=aae15985ebae4792b7f74e6f31e8ac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.745998 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_546234ae53e94032967f721a5e695354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:28.746376 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3b07b2e0284761a96ea3650506638b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.746454 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_546234ae53e94032967f721a5e695354 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.750983 892 tablet_service.cc:1468] Processing CreateTablet for tablet 6ab9a28007e74d9cbd1c7affbf5c660c (DEFAULT_TABLE table=default.Table_dd10ce6555604cb084e1b1d299cf9aa2 [id=d30ee521df404093a399d5bf600cf3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.751356 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab9a28007e74d9cbd1c7affbf5c660c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.755250 891 tablet_service.cc:1468] Processing CreateTablet for tablet 5986bb52335c46bf845c48cb044495b1 (DEFAULT_TABLE table=default.Table_5ddc30bfdeea430e8109fea6979694a5 [id=2307d739c95a4d0ca4efe1d533f65545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.755591 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5986bb52335c46bf845c48cb044495b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.759433 754 tablet_service.cc:1468] Processing CreateTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 (DEFAULT_TABLE table=default.Table_ce15eace6bc5411d808b723c797ed21e [id=8d4a8846aa4f4826880a1ed7fe06ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.759496 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.759903 749 tablet_service.cc:1468] Processing CreateTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.760910 741 tablet_service.cc:1468] Processing CreateTablet for tablet 683954dc7ecc4185847f46a05fd2c53d (DEFAULT_TABLE table=default.Table_b527d528908b40de9346966341c2f3fb [id=d091d89c64f840278a5634306587411d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.762761 1162 ts_tablet_manager.cc:1428] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20250625 07:52:28.764217 1162 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.765574 1162 tablet_bootstrap.cc:654] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.767611 1163 tablet_bootstrap.cc:700] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Time spent opening tablet: real 0.114s user 0.000s sys 0.001s
I20250625 07:52:28.767957 1163 tablet_bootstrap.cc:654] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.768630 1977 raft_consensus.cc:491] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.768826 1977 raft_consensus.cc:513] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.769585 1977 leader_election.cc:290] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.769804 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1afe90251c964e369d05e82c7db19326" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:28.770256 851 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:28.771684 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 813f9f9fccc34685b73847c9d590ceb6 consensus state has no leader
I20250625 07:52:28.772837 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 (DEFAULT_TABLE table=default.Table_4e2dc27726d944c5885d62ed4258acfc [id=0779497b504d4295938af3db616b1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.773273 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d34bfe377ba421daede5ae68848a2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.773761 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d14c6b22d6b745c9aadedb8f7725f4b4" } modify_external_catalogs: true
I20250625 07:52:28.773922 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d14c6b22d6b745c9aadedb8f7725f4b4" } modify_external_catalogs: true
I20250625 07:52:28.779637 882 tablet_service.cc:1468] Processing CreateTablet for tablet 81cd2b2215cd4a17add2306d01d99b9a (DEFAULT_TABLE table=default.Table_5a68ddb21adc44d584bf839cb84d1c18 [id=687ef996cfaf4f1787d709d3ff0fcccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.780046 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cd2b2215cd4a17add2306d01d99b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.782755 1166 ts_tablet_manager.cc:1918] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.782860 1166 log.cc:1199] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:28.783162 1166 ts_tablet_manager.cc:1939] T 366d3e6fcc9d49be8fbd2cd070632861 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.783200 750 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.784514 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656630686bfb49a59c5e7fc8a5cfb4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.784703 885 tablet_service.cc:1468] Processing CreateTablet for tablet e34e54db2181402dab9b6c5b22bc7da9 (DEFAULT_TABLE table=default.Table_fa33440d4d5740c38f6730f962447c3e [id=b07e7cda7a814f57a6aa51edc0d049c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.785032 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34e54db2181402dab9b6c5b22bc7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.785040 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 366d3e6fcc9d49be8fbd2cd070632861 (table default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]) successfully deleted
W20250625 07:52:28.785344 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fdad7dc1b27e4d87a747642a6d47230c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdad7dc1b27e4d87a747642a6d47230c already in progress: creating tablet
I20250625 07:52:28.788879 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.789880 894 tablet_service.cc:1468] Processing CreateTablet for tablet 07f0161046bd42cbb3ecbcf004c13891 (DEFAULT_TABLE table=default.Table_b2e47db078dd4a8c8ec4c8df33955362 [id=cb56a7d28b144e2884b80c21be1b3416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.790256 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f0161046bd42cbb3ecbcf004c13891. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.794219 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7b5cf53827f74c389a995bd409c38087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.794839 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b5cf53827f74c389a995bd409c38087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.795580 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6ddaef2306f14d4ba08e26ee4d1822b6" } new_table_name: "default.Table_fba333af0c074b57b6fefee2fb861637" modify_external_catalogs: true
I20250625 07:52:28.796754 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b98c1f91e4914fb29c6b413e92cb1929
I20250625 07:52:28.796949 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683954dc7ecc4185847f46a05fd2c53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.797272 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b99fa339b63429ca65b10e0fadd2fa3" } modify_external_catalogs: true
I20250625 07:52:28.797430 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b99fa339b63429ca65b10e0fadd2fa3" } modify_external_catalogs: true
I20250625 07:52:28.797641 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_226f128a33ac44438af91af68b3859b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.798089 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_226f128a33ac44438af91af68b3859b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.799412 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a559f40e04b9464ab39d21bf2fdd6ac6" } new_table_name: "default.Table_f9f4b30d1e6049428f31ff5791dcc3a0" modify_external_catalogs: true
I20250625 07:52:28.800529 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 47c9aca996884ffeab88381dd3cb189b
I20250625 07:52:28.800652 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 (DEFAULT_TABLE table=default.Table_1a052ae2f0774e85bdc42172e61770d1 [id=1bad4ceb6f1642da98e2aa014e78a1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.800884 893 tablet_service.cc:1468] Processing CreateTablet for tablet cd0b0642010a43f5a81748b7682f6a67 (DEFAULT_TABLE table=default.Table_98f0756428904e40a7406141b5194756 [id=dd5b69cfe373405d9b1acff25a920e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.801041 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3eafac1b2b46acb2ac6fca087b7f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.801189 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0b0642010a43f5a81748b7682f6a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.802021 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a80d81cc2d524f318b858a9bd97a89b8
I20250625 07:52:28.804278 879 tablet_service.cc:1468] Processing CreateTablet for tablet 448a7d67fc014411ab294b19c65637ee (DEFAULT_TABLE table=default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8 [id=38cb713de33f4852a9930e19c378c20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.804435 888 tablet_service.cc:1468] Processing CreateTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e (DEFAULT_TABLE table=default.Table_602ca03b4f824a7686588dc776b8bed6 [id=91d401787d5d4f09948941ecc4b1a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.804586 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448a7d67fc014411ab294b19c65637ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.804774 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113c05d81d2943e38bcd62bf83a46a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.807982 876 tablet_service.cc:1468] Processing CreateTablet for tablet f406d1c8777f474c97f6e804cb230bb6 (DEFAULT_TABLE table=default.Table_006e977b28ff4dbe9f2091ba6f1879e2 [id=0621b19752f041429ae5533ce4f57566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.808291 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406d1c8777f474c97f6e804cb230bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.809388 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.809854 886 tablet_service.cc:1468] Processing CreateTablet for tablet f8327a1aeb2743b3993ffac98b6ca2c0 (DEFAULT_TABLE table=default.Table_ca8eeef7078a4df39792b710bafcbd22 [id=a1c720dea2684e9c8e5192084e88686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.810175 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8327a1aeb2743b3993ffac98b6ca2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.812975 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:28.813112 880 tablet_service.cc:1468] Processing CreateTablet for tablet b45add1fbcc2423d92b7b0001f946cb6 (DEFAULT_TABLE table=default.Table_e2e1925e33414e199b380b98a0c6e96f [id=6cd6d9a3c91a44e7a8d454777677d909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.813597 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45add1fbcc2423d92b7b0001f946cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.814599 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ae8736f8b7954519bedb73c2427040e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.815112 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae8736f8b7954519bedb73c2427040e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.818228 1166 tablet_replica.cc:331] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.818498 1166 raft_consensus.cc:2241] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.818720 1166 raft_consensus.cc:2270] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.822117 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 450727d8f33e4eb489969b7974cdeeb1 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:28.818907 754 tablet_service.cc:1515] Processing DeleteTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.829335 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
W20250625 07:52:28.831199 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1a653e84823d467f9beb897f2bfb3b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a653e84823d467f9beb897f2bfb3b3b already in progress: creating tablet
W20250625 07:52:28.831552 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 51744ae4b84e47cea49fb54023777a41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51744ae4b84e47cea49fb54023777a41 already in progress: creating tablet
W20250625 07:52:28.831817 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4243f6f8c1154fc89b3aa1874441b98d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f6f8c1154fc89b3aa1874441b98d already in progress: creating tablet
W20250625 07:52:28.832046 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 381ae5ef5df9437d9bfe0a51987dfe62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 381ae5ef5df9437d9bfe0a51987dfe62 already in progress: creating tablet
I20250625 07:52:28.833447 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c9aca996884ffeab88381dd3cb189b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:28.840170 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 4 got 0
I20250625 07:52:28.841926 1162 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.842114 1162 ts_tablet_manager.cc:1397] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20250625 07:52:28.842841 1162 raft_consensus.cc:357] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.843240 1162 raft_consensus.cc:383] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.843366 1162 raft_consensus.cc:738] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.843647 1162 consensus_queue.cc:260] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.845633 1166 ts_tablet_manager.cc:1905] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.846354 1162 ts_tablet_manager.cc:1428] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:28.846467 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:28.846642 1162 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.848166 1162 tablet_bootstrap.cc:654] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.850420 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:28.852986 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 912) took 1270 ms. Trace:
I20250625 07:52:28.853132 1160 rpcz_store.cc:276] 0625 07:52:27.582781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.584978 (+ 2197us) service_pool.cc:225] Handling call
0625 07:52:28.852023 (+1267045us) ts_tablet_manager.cc:1145] Deleting tablet 231ced6803944e24829caf7853da017d
0625 07:52:28.852849 (+ 826us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.852967 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266924}
I20250625 07:52:28.855803 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:28.856068 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 905) took 1370 ms. Trace:
I20250625 07:52:28.856228 1160 rpcz_store.cc:276] 0625 07:52:27.485773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.565194 (+ 79421us) service_pool.cc:225] Handling call
0625 07:52:28.853560 (+1288366us) ts_tablet_manager.cc:1145] Deleting tablet 5de0c5bf60b24808af124c95833d9c25
0625 07:52:28.855956 (+ 2396us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.856050 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248774}
I20250625 07:52:28.856568 1160 tablet_replica.cc:331] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:28.856799 1160 raft_consensus.cc:2241] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.857012 1160 raft_consensus.cc:2270] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.857539 1160 ts_tablet_manager.cc:1905] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.859062 881 tablet_service.cc:1468] Processing CreateTablet for tablet 190c4bfe74344994babdc7e94513e6c2 (DEFAULT_TABLE table=default.Table_90f97f84512542b29acecb9db3e1172d [id=0fd4ed98e556473292ed565b2a60ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.860224 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5755b609b94f446f83341c16870ac1c1 (DEFAULT_TABLE table=default.Table_6aceb5dc89a54074b4b33990008cb85e [id=621ed69e94d24dc28714b0ce7ae93987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.860613 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190c4bfe74344994babdc7e94513e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.861723 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5755b609b94f446f83341c16870ac1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.868806 1161 log.cc:1199] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/29eaa8d93e5b4d1d8ed147b60c7da859
I20250625 07:52:28.869416 1161 ts_tablet_manager.cc:1939] T 29eaa8d93e5b4d1d8ed147b60c7da859 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:28.869827 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 876) took 1271 ms. Trace:
I20250625 07:52:28.870028 1161 rpcz_store.cc:276] 0625 07:52:27.598129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.598182 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:28.663377 (+1065195us) ts_tablet_manager.cc:1145] Deleting tablet 29eaa8d93e5b4d1d8ed147b60c7da859
0625 07:52:28.663385 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.700882 (+ 37497us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.703565 (+ 2683us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.869794 (+166229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064961}
I20250625 07:52:28.870497 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 877) took 1222 ms. Trace:
I20250625 07:52:28.870636 1161 rpcz_store.cc:276] 0625 07:52:27.647845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.647895 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:28.870369 (+1222474us) ts_tablet_manager.cc:1145] Deleting tablet 4f1c150fb54c405c8dc61ac133d98b13
0625 07:52:28.870377 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.870467 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222296}
I20250625 07:52:28.870956 1161 tablet_replica.cc:331] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:28.871201 1161 raft_consensus.cc:2241] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.871419 1161 raft_consensus.cc:2270] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.873004 890 tablet_service.cc:1468] Processing CreateTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 (DEFAULT_TABLE table=default.Table_8319e385441549e98fc4f6f600dd17b7 [id=5a3e037b6b7944968b285f678c967d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.873034 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 64a5ff005b424ae5ae9831263abf58ef
I20250625 07:52:28.873456 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b1ec100ca4488c9d7fc470246ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.874351 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33f56c267457451e864e277e110c68fa consensus state has no leader
I20250625 07:52:28.874673 2003 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:28.876894 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ddf70e7ccb3644f4b28ceb97decd744a" } new_table_name: "default.Table_475fbc5c9adc457c997ddcc8fc2ad236" modify_external_catalogs: true
I20250625 07:52:28.877040 885 tablet_service.cc:1468] Processing CreateTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 (DEFAULT_TABLE table=default.Table_c3197b8d0402490b86e0fbc10c91c62e [id=aee83d7d1b9a4ac7aac0d19123403db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.877351 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fa6e5ac74444729645a27fc2945bc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.878193 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20750adfe4c14c55bddc2ea3bbf99ef2 consensus state has no leader
I20250625 07:52:28.710179 1011 tablet_service.cc:1468] Processing CreateTablet for tablet de3b07b2e0284761a96ea3650506638b (DEFAULT_TABLE table=default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a [id=aae15985ebae4792b7f74e6f31e8ac9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.880589 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de3b07b2e0284761a96ea3650506638b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.745342 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f4359fd842a4760bae75169e8914f1c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.750766 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6ab9a28007e74d9cbd1c7affbf5c660c (DEFAULT_TABLE table=default.Table_dd10ce6555604cb084e1b1d299cf9aa2 [id=d30ee521df404093a399d5bf600cf3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.885793 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab9a28007e74d9cbd1c7affbf5c660c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.753391 1976 raft_consensus.cc:491] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:28.888967 850 leader_election.cc:343] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.888936 1976 raft_consensus.cc:513] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.889079 850 leader_election.cc:304] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.889508 1977 raft_consensus.cc:2747] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.889590 1976 leader_election.cc:290] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:28.890000 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:28.755066 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 5986bb52335c46bf845c48cb044495b1 (DEFAULT_TABLE table=default.Table_5ddc30bfdeea430e8109fea6979694a5 [id=2307d739c95a4d0ca4efe1d533f65545]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.769913 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1afe90251c964e369d05e82c7db19326" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.772111 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 (DEFAULT_TABLE table=default.Table_4e2dc27726d944c5885d62ed4258acfc [id=0779497b504d4295938af3db616b1c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.778743 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 81cd2b2215cd4a17add2306d01d99b9a (DEFAULT_TABLE table=default.Table_5a68ddb21adc44d584bf839cb84d1c18 [id=687ef996cfaf4f1787d709d3ff0fcccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.784051 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e34e54db2181402dab9b6c5b22bc7da9 (DEFAULT_TABLE table=default.Table_fa33440d4d5740c38f6730f962447c3e [id=b07e7cda7a814f57a6aa51edc0d049c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.789134 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 07f0161046bd42cbb3ecbcf004c13891 (DEFAULT_TABLE table=default.Table_b2e47db078dd4a8c8ec4c8df33955362 [id=cb56a7d28b144e2884b80c21be1b3416]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.796680 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 (DEFAULT_TABLE table=default.Table_1a052ae2f0774e85bdc42172e61770d1 [id=1bad4ceb6f1642da98e2aa014e78a1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.797415 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 448a7d67fc014411ab294b19c65637ee (DEFAULT_TABLE table=default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8 [id=38cb713de33f4852a9930e19c378c20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.798066 1015 tablet_service.cc:1468] Processing CreateTablet for tablet f406d1c8777f474c97f6e804cb230bb6 (DEFAULT_TABLE table=default.Table_006e977b28ff4dbe9f2091ba6f1879e2 [id=0621b19752f041429ae5533ce4f57566]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.798697 1019 tablet_service.cc:1468] Processing CreateTablet for tablet cd0b0642010a43f5a81748b7682f6a67 (DEFAULT_TABLE table=default.Table_98f0756428904e40a7406141b5194756 [id=dd5b69cfe373405d9b1acff25a920e3f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:28.890810 985 leader_election.cc:343] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 81976db0a2f94128a89842a12f067231
I20250625 07:52:28.799639 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e (DEFAULT_TABLE table=default.Table_602ca03b4f824a7686588dc776b8bed6 [id=91d401787d5d4f09948941ecc4b1a918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.890920 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5986bb52335c46bf845c48cb044495b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.800062 1013 tablet_service.cc:1468] Processing CreateTablet for tablet b45add1fbcc2423d92b7b0001f946cb6 (DEFAULT_TABLE table=default.Table_e2e1925e33414e199b380b98a0c6e96f [id=6cd6d9a3c91a44e7a8d454777677d909]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.800318 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 (DEFAULT_TABLE table=default.Table_8319e385441549e98fc4f6f600dd17b7 [id=5a3e037b6b7944968b285f678c967d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.801540 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f8327a1aeb2743b3993ffac98b6ca2c0 (DEFAULT_TABLE table=default.Table_ca8eeef7078a4df39792b710bafcbd22 [id=a1c720dea2684e9c8e5192084e88686c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.891361 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3eafac1b2b46acb2ac6fca087b7f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.891669 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 64a5ff005b424ae5ae9831263abf58ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:28.895044 850 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.895154 850 leader_election.cc:304] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:28.895354 986 leader_election.cc:343] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 81976db0a2f94128a89842a12f067231
I20250625 07:52:28.895496 986 leader_election.cc:304] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:28.896862 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 448a7d67fc014411ab294b19c65637ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.897269 1976 raft_consensus.cc:2747] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.899945 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e609e7267a66453dab2b666d69aebd02 (DEFAULT_TABLE table=default.Table_f62f60f959ba405b8f9f91731541cee3 [id=0c11b58976464c94967abf87773d89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.858422 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 190c4bfe74344994babdc7e94513e6c2 (DEFAULT_TABLE table=default.Table_90f97f84512542b29acecb9db3e1172d [id=0fd4ed98e556473292ed565b2a60ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.859668 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 5755b609b94f446f83341c16870ac1c1 (DEFAULT_TABLE table=default.Table_6aceb5dc89a54074b4b33990008cb85e [id=621ed69e94d24dc28714b0ce7ae93987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.905992 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e127e5950e034aed9b6b4ec3f6e1af72" } new_table_name: "default.Table_a4acbad3c3ab41a4896fd9a719d52175" modify_external_catalogs: true
I20250625 07:52:28.906401 1977 raft_consensus.cc:2747] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.906585 1977 raft_consensus.cc:491] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.906682 1977 raft_consensus.cc:513] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.907356 1977 leader_election.cc:290] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.907596 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:28.907656 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:28.908089 850 leader_election.cc:343] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:28.908179 851 leader_election.cc:343] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.908237 850 leader_election.cc:304] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.908542 1977 raft_consensus.cc:2747] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.909485 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f7ccc476abc14686bbffff9afc893419"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:28.910022 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7ccc476abc14686bbffff9afc893419 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:28.910667 1161 ts_tablet_manager.cc:1905] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.705917 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ec2459d73e594a0c9622acc14bf1c5b5 (DEFAULT_TABLE table=default.Table_f39da4a74c09429db8be207816d8f5d1 [id=5a1b45d98e574363a7cc95a244a93d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.914341 1166 ts_tablet_manager.cc:1918] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.915546 1166 log.cc:1199] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:28.916028 1166 ts_tablet_manager.cc:1939] T a33b7cab19244ec08d3821b6cc4c072d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:28.916973 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a33b7cab19244ec08d3821b6cc4c072d (table default.Table_2f89a55bc2694564b94bb87f42761d2a [id=dff8e8b2f6da4c39b78f536bcc8467e5]) successfully deleted
I20250625 07:52:28.917335 1166 tablet_replica.cc:331] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:28.917747 1166 raft_consensus.cc:2241] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:28.918187 1166 raft_consensus.cc:2270] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:28.921286 1166 ts_tablet_manager.cc:1905] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:28.928053 2002 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:28.935393 1976 raft_consensus.cc:491] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.935576 1976 raft_consensus.cc:513] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.936239 1976 leader_election.cc:290] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:28.937167 985 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e41e53efb6f342d2b1e4d642f7aaad10
W20250625 07:52:28.937621 986 leader_election.cc:343] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: e41e53efb6f342d2b1e4d642f7aaad10
I20250625 07:52:28.937803 986 leader_election.cc:304] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:28.938316 1976 raft_consensus.cc:2747] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.853277 1158 tablet_bootstrap.cc:492] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:28.943182 1166 ts_tablet_manager.cc:1918] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.945272 1977 raft_consensus.cc:491] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.945418 1977 raft_consensus.cc:513] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:28.946103 1977 leader_election.cc:290] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.946456 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e289ea5553ac4ff983871ece4fc72ee5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:28.951689 1166 log.cc:1199] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b1422987deca41a79f8db666d6961643
I20250625 07:52:28.952054 1166 ts_tablet_manager.cc:1939] T b1422987deca41a79f8db666d6961643 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:28.952284 851 leader_election.cc:343] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.954181 1162 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.954371 1162 ts_tablet_manager.cc:1397] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.108s user 0.001s sys 0.008s
W20250625 07:52:28.954491 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a80d81cc2d524f318b858a9bd97a89b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80d81cc2d524f318b858a9bd97a89b8 already in progress: creating tablet
I20250625 07:52:28.955243 1162 raft_consensus.cc:357] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:28.955377 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b98c1f91e4914fb29c6b413e92cb1929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98c1f91e4914fb29c6b413e92cb1929 already in progress: creating tablet
I20250625 07:52:28.955580 1162 raft_consensus.cc:383] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.955710 1162 raft_consensus.cc:738] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.955998 1162 consensus_queue.cc:260] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:28.956117 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 47c9aca996884ffeab88381dd3cb189b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c9aca996884ffeab88381dd3cb189b already in progress: creating tablet
W20250625 07:52:28.958590 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 64a5ff005b424ae5ae9831263abf58ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a5ff005b424ae5ae9831263abf58ef already in progress: creating tablet
I20250625 07:52:28.960384 1162 ts_tablet_manager.cc:1428] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250625 07:52:28.965751 1162 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.966063 1856 raft_consensus.cc:491] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:28.966176 1856 raft_consensus.cc:513] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.966686 1856 leader_election.cc:290] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:28.966188 1989 sys_catalog.cc:263] Verifying existing consensus state
W20250625 07:52:28.967959 713 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:28.969800 1162 tablet_bootstrap.cc:654] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.880155 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 (DEFAULT_TABLE table=default.Table_c3197b8d0402490b86e0fbc10c91c62e [id=aee83d7d1b9a4ac7aac0d19123403db1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:28.901225 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f406d1c8777f474c97f6e804cb230bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.978498 1162 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:28.978631 1162 ts_tablet_manager.cc:1397] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20250625 07:52:28.979346 1162 raft_consensus.cc:357] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.979496 1162 raft_consensus.cc:383] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:28.979544 1162 raft_consensus.cc:738] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:28.979755 1162 consensus_queue.cc:260] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:28.980546 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fe9a633f1a8e450ba2a6db9adc198734" } new_table_name: "default.Table_cf4e82b97e4649fc8e50c62a18f22d20" modify_external_catalogs: true
I20250625 07:52:28.946833 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e289ea5553ac4ff983871ece4fc72ee5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:28.981405 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fa6e5ac74444729645a27fc2945bc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:28.981462 850 leader_election.cc:343] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:28.981581 850 leader_election.cc:304] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:28.981956 1977 raft_consensus.cc:2747] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:28.984354 1162 ts_tablet_manager.cc:1428] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250625 07:52:28.987864 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f0161046bd42cbb3ecbcf004c13891. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.988196 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e609e7267a66453dab2b666d69aebd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.988664 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e34e54db2181402dab9b6c5b22bc7da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.989233 1162 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:28.991641 1162 tablet_bootstrap.cc:654] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:28.992719 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b45add1fbcc2423d92b7b0001f946cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.996830 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 113c05d81d2943e38bcd62bf83a46a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:28.998960 1161 ts_tablet_manager.cc:1918] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:28.999182 1161 log.cc:1199] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a777f64425384be5bccc86b197626618
I20250625 07:52:29.000828 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d34bfe377ba421daede5ae68848a2d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.001188 1977 raft_consensus.cc:491] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.001343 1977 raft_consensus.cc:513] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.002091 1977 leader_election.cc:290] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.002676 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0929ed76622248d1a41f08f4258108cd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:29.002745 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3ddaab806779471489af8e38d15a75d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.003329 851 leader_election.cc:343] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:29.003389 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ddaab806779471489af8e38d15a75d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.003422 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0929ed76622248d1a41f08f4258108cd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:29.003829 850 leader_election.cc:343] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.003908 850 leader_election.cc:304] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.004259 1977 raft_consensus.cc:2747] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.004777 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_da2cc3d0fc2a4550847f4606db8c777a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.005317 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da2cc3d0fc2a4550847f4606db8c777a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.006367 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6c72c34aa1954d40b2be6b4988c05f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.006799 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0b0642010a43f5a81748b7682f6a67. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:29.006850 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c72c34aa1954d40b2be6b4988c05f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.007898 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e5cc8a2851a74b148a50729e9402a687" } modify_external_catalogs: true
I20250625 07:52:29.008041 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e5cc8a2851a74b148a50729e9402a687" } modify_external_catalogs: true
I20250625 07:52:29.011593 1161 ts_tablet_manager.cc:1939] T a777f64425384be5bccc86b197626618 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:28.962627 1976 raft_consensus.cc:491] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.011911 1976 raft_consensus.cc:513] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.012100 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 878) took 1348 ms. Trace:
I20250625 07:52:29.012200 1161 rpcz_store.cc:276] 0625 07:52:27.663579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.663633 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:28.870896 (+1207263us) ts_tablet_manager.cc:1145] Deleting tablet a777f64425384be5bccc86b197626618
0625 07:52:28.870905 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.987718 (+116813us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:28.998840 (+ 11122us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.012048 (+ 13208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207088}
I20250625 07:52:29.012682 1161 tablet_replica.cc:331] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.012756 1976 leader_election.cc:290] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:28.968057 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba0735d576f947d395bee8d1b865f2fd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.012902 1161 raft_consensus.cc:2241] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.013090 1161 raft_consensus.cc:2270] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.011641 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cd2b2215cd4a17add2306d01d99b9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:29.013644 985 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 875fa1b46903452eaed1f2235d63dfdf
W20250625 07:52:29.013609 713 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.013756 713 leader_election.cc:304] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.014626 1856 raft_consensus.cc:2747] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:29.015334 986 leader_election.cc:343] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 875fa1b46903452eaed1f2235d63dfdf
I20250625 07:52:29.015443 986 leader_election.cc:304] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.015779 1976 raft_consensus.cc:2747] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.016819 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b1ec100ca4488c9d7fc470246ffe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.024827 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8327a1aeb2743b3993ffac98b6ca2c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.040347 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5755b609b94f446f83341c16870ac1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.044834 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 190c4bfe74344994babdc7e94513e6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.046219 1989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap starting.
I20250625 07:52:28.706849 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1488eddcbbc84562a2bcb73099161de8 (DEFAULT_TABLE table=default.Table_2166d8d523c34441943a1d577b745b4d [id=452b083a9563462ba0bcf108fcec8ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.049101 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1488eddcbbc84562a2bcb73099161de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.049193 1161 ts_tablet_manager.cc:1905] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.056376 1158 ts_tablet_manager.cc:1397] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.401s user 0.009s sys 0.003s
I20250625 07:52:29.061548 1158 raft_consensus.cc:357] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.061746 1158 raft_consensus.cc:383] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.061829 1158 raft_consensus.cc:738] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.062050 1158 consensus_queue.cc:260] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.062556 1158 ts_tablet_manager.cc:1428] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.000s sys 0.000s
I20250625 07:52:29.062858 1158 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.064746 1158 tablet_bootstrap.cc:654] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.073078 1856 raft_consensus.cc:491] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.073300 1856 raft_consensus.cc:513] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.074132 1856 leader_election.cc:290] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.074695 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2459d73e594a0c9622acc14bf1c5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.074982 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:29.075228 912 raft_consensus.cc:2466] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:29.075727 713 leader_election.cc:304] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:29.076308 1856 raft_consensus.cc:2802] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:29.076395 1856 raft_consensus.cc:491] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:29.076365 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.076455 1856 raft_consensus.cc:3058] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:29.077997 1163 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.078148 1163 ts_tablet_manager.cc:1397] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.453s user 0.000s sys 0.003s
I20250625 07:52:29.078878 1163 raft_consensus.cc:357] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.079018 1163 raft_consensus.cc:383] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.079061 1163 raft_consensus.cc:738] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.079282 1163 consensus_queue.cc:260] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.079373 1856 raft_consensus.cc:513] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.079720 1163 ts_tablet_manager.cc:1428] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:29.080175 1163 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.081481 1163 tablet_bootstrap.cc:654] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.085479 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:29.085657 912 raft_consensus.cc:3058] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:29.091917 713 leader_election.cc:343] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.093063 1161 ts_tablet_manager.cc:1918] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.093228 1161 log.cc:1199] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b377c8111c9f4ac2ba05e65694d611b9
W20250625 07:52:29.093405 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5582df28a7d54fe8ad16b2f8ea1edabb consensus state has no leader
I20250625 07:52:29.093575 1161 ts_tablet_manager.cc:1939] T b377c8111c9f4ac2ba05e65694d611b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.094134 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 879) took 1399 ms. Trace:
I20250625 07:52:29.094247 1161 rpcz_store.cc:276] 0625 07:52:27.694707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.696537 (+ 1830us) service_pool.cc:225] Handling call
0625 07:52:29.012613 (+1316076us) ts_tablet_manager.cc:1145] Deleting tablet b377c8111c9f4ac2ba05e65694d611b9
0625 07:52:29.012621 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.052079 (+ 39458us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.092934 (+ 40855us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.094085 (+ 1151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315838}
I20250625 07:52:29.094599 1161 tablet_replica.cc:331] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.094727 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.094791 1161 raft_consensus.cc:2241] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.094954 1161 raft_consensus.cc:2270] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.095181 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c (DEFAULT_TABLE table=default.Table_895827fb99994402b75be290df98a2ff [id=e8069108ae2b4dd392d7253ae8e6d4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.095577 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.096021 890 tablet_service.cc:1468] Processing CreateTablet for tablet e609e7267a66453dab2b666d69aebd02 (DEFAULT_TABLE table=default.Table_f62f60f959ba405b8f9f91731541cee3 [id=0c11b58976464c94967abf87773d89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.096362 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e609e7267a66453dab2b666d69aebd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.098527 887 tablet_service.cc:1468] Processing CreateTablet for tablet 65caf04d39a044ec82514f309128e767 (DEFAULT_TABLE table=default.Table_09bebf0eae58464c970e326392341524 [id=d90831797f054efcb9b5526e6d3c2cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.098875 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65caf04d39a044ec82514f309128e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.100997 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.101415 876 tablet_service.cc:1468] Processing CreateTablet for tablet b85adedf7d2c45a1b33ee4a5633ee734 (DEFAULT_TABLE table=default.Table_3b5acb6fb6cd40d2855892e2d28bed37 [id=d87af998b5ff47d69efb9780b11feaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.101742 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85adedf7d2c45a1b33ee4a5633ee734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.103729 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 381ae5ef5df9437d9bfe0a51987dfe62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.103822 1976 raft_consensus.cc:491] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.104008 1976 raft_consensus.cc:513] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.104842 1976 leader_election.cc:290] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:29.105998 986 leader_election.cc:343] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6606a9559e6b47d28669083af5aee5bd
I20250625 07:52:29.106040 893 tablet_service.cc:1468] Processing CreateTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 (DEFAULT_TABLE table=default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.106360 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dff56bc7445abcd6b7c1cb525553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.112438 912 raft_consensus.cc:2466] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:29.112622 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.113035 713 leader_election.cc:304] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:29.113305 879 tablet_service.cc:1468] Processing CreateTablet for tablet bff347d838fe4e56a195a957d0deb742 (DEFAULT_TABLE table=default.Table_2f8acc41371a4800a82de306adce74db [id=b6406562b9b14056ba9b938c38e2fe89]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:29.113474 985 leader_election.cc:343] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6606a9559e6b47d28669083af5aee5bd
I20250625 07:52:29.113672 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff347d838fe4e56a195a957d0deb742. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.113734 985 leader_election.cc:304] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.114228 1976 raft_consensus.cc:2747] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.115109 1856 leader_election.cc:290] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.115193 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:29.115339 1856 raft_consensus.cc:2802] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:29.115842 1856 raft_consensus.cc:695] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:29.116235 1856 consensus_queue.cc:237] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:29.118019 713 leader_election.cc:343] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.119148 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 875fa1b46903452eaed1f2235d63dfdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.121791 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:29.127022 713 consensus_peers.cc:487] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:29.131101 1029 tablet_service.cc:1468] Processing CreateTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 (DEFAULT_TABLE table=default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.131685 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab2dff56bc7445abcd6b7c1cb525553. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.137056 1016 tablet_service.cc:1468] Processing CreateTablet for tablet b85adedf7d2c45a1b33ee4a5633ee734 (DEFAULT_TABLE table=default.Table_3b5acb6fb6cd40d2855892e2d28bed37 [id=d87af998b5ff47d69efb9780b11feaa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.137359 1162 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.137516 1162 ts_tablet_manager.cc:1397] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.148s user 0.019s sys 0.043s
I20250625 07:52:29.137743 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85adedf7d2c45a1b33ee4a5633ee734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.138365 1162 raft_consensus.cc:357] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.138536 1162 raft_consensus.cc:383] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.138588 1162 raft_consensus.cc:738] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.138829 1162 consensus_queue.cc:260] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.139307 1162 ts_tablet_manager.cc:1428] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:29.139744 1161 ts_tablet_manager.cc:1905] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.144837 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a653e84823d467f9beb897f2bfb3b3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.145380 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 65caf04d39a044ec82514f309128e767 (DEFAULT_TABLE table=default.Table_09bebf0eae58464c970e326392341524 [id=d90831797f054efcb9b5526e6d3c2cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.145820 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65caf04d39a044ec82514f309128e767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.148834 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 4243f6f8c1154fc89b3aa1874441b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.149338 1013 tablet_service.cc:1468] Processing CreateTablet for tablet bff347d838fe4e56a195a957d0deb742 (DEFAULT_TABLE table=default.Table_2f8acc41371a4800a82de306adce74db [id=b6406562b9b14056ba9b938c38e2fe89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.149778 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff347d838fe4e56a195a957d0deb742. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.157456 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 51744ae4b84e47cea49fb54023777a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.157768 881 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.158160 1162 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.158258 1976 raft_consensus.cc:491] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.158401 1976 raft_consensus.cc:513] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.159037 1976 leader_election.cc:290] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:29.160641 986 leader_election.cc:343] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6463d2bc39034ee7ad1b8bfcfcb4b0fe
W20250625 07:52:29.160980 985 leader_election.cc:343] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:29.161078 985 leader_election.cc:304] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.161634 1976 raft_consensus.cc:2747] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.163203 1163 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.163352 1163 ts_tablet_manager.cc:1397] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.000s
I20250625 07:52:29.164120 1163 raft_consensus.cc:357] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.164296 1163 raft_consensus.cc:383] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.164357 1163 raft_consensus.cc:738] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.164603 1163 consensus_queue.cc:260] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.165264 1163 ts_tablet_manager.cc:1428] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:29.168226 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c (DEFAULT_TABLE table=default.Table_895827fb99994402b75be290df98a2ff [id=e8069108ae2b4dd392d7253ae8e6d4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.168596 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2aa7a64bfd49bc8e3accf840c66a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.169620 1163 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.170751 1158 tablet_bootstrap.cc:492] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.170889 1158 ts_tablet_manager.cc:1397] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.108s user 0.001s sys 0.001s
I20250625 07:52:29.171306 1163 tablet_bootstrap.cc:654] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.171627 1158 raft_consensus.cc:357] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.171777 1158 raft_consensus.cc:383] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.171831 1158 raft_consensus.cc:738] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.174322 1162 tablet_bootstrap.cc:654] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.178866 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_65094aac02f94466a02285f60f743c29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.179428 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65094aac02f94466a02285f60f743c29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.179602 1158 consensus_queue.cc:260] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.180157 1158 ts_tablet_manager.cc:1428] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 07:52:29.182590 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 51744ae4b84e47cea49fb54023777a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:29.182814 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 88383.8us
I20250625 07:52:29.183573 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 5c28a064006145eea000e74da10d0794 (table default.Table_9bbce7454a884554b31faffc79a33b99 [id=cbb5aafb8a9243d98c7d8578865c13b9]) successfully deleted
W20250625 07:52:29.183635 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7f82e83ed924d2bbedd33d96642a10a consensus state has no leader
W20250625 07:52:29.183801 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 231ced6803944e24829caf7853da017d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 231ced6803944e24829caf7853da017d already in progress: creating tablet
I20250625 07:52:29.184017 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]) successfully deleted
I20250625 07:52:29.184435 1158 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:29.184219 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a396bde4b5964697ad0090eab777ba0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a396bde4b5964697ad0090eab777ba0e already in progress: creating tablet
W20250625 07:52:29.185385 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5fec0e5f364ca093879a8ad2e0bcfc already in progress: creating tablet
I20250625 07:52:29.185765 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 29eaa8d93e5b4d1d8ed147b60c7da859 (table default.Table_97384ebbc1584955af0a6bc425084522 [id=131a7d8cbeb34a0a8db53be305d31ccd]) successfully deleted
W20250625 07:52:29.186012 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4f1c150fb54c405c8dc61ac133d98b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1c150fb54c405c8dc61ac133d98b13 already in progress: creating tablet
I20250625 07:52:29.186564 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
I20250625 07:52:29.186606 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a777f64425384be5bccc86b197626618 (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
I20250625 07:52:29.186754 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:29.188092 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b377c8111c9f4ac2ba05e65694d611b9 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]) successfully deleted
I20250625 07:52:29.188192 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_65740b2f1a624011b8d0248cb3bb723e" } modify_external_catalogs: true
I20250625 07:52:29.188194 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_7ab4243dccfa40bda777a9a70016ec4f [id=0e236158e41f4ea5aef616e7777bb94f] alter complete (version 4)
I20250625 07:52:29.188410 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_65740b2f1a624011b8d0248cb3bb723e" } modify_external_catalogs: true
I20250625 07:52:29.189261 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_864fa9594e1d4f97aef0122611c515cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.189821 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_864fa9594e1d4f97aef0122611c515cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.190795 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4709f33f040149c49bd273524c9fd4f0 (DEFAULT_TABLE table=default.Table_71d91cadf45d4ae1a05a5db26871f709 [id=9bc20b9bd6a14d85bba8973cf718df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.191533 877 tablet_service.cc:1468] Processing CreateTablet for tablet 4709f33f040149c49bd273524c9fd4f0 (DEFAULT_TABLE table=default.Table_71d91cadf45d4ae1a05a5db26871f709 [id=9bc20b9bd6a14d85bba8973cf718df94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.191905 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4709f33f040149c49bd273524c9fd4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.192070 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2231a0c930064911a051c1cf00c3d2e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.192523 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2231a0c930064911a051c1cf00c3d2e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.195628 755 tablet_service.cc:1468] Processing CreateTablet for tablet 499473d85b284d6cb773148ab276992e (DEFAULT_TABLE table=default.Table_34b97f57ddc44127b6e68fb8a5542a1c [id=07382324ce9f4c31bbc01fb29c076258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.196066 753 tablet_service.cc:1468] Processing CreateTablet for tablet 1b9f10a26894418795f1c928292afd3d (DEFAULT_TABLE table=default.Table_b80453878cf143caba5ec40c14087313 [id=a3c34c18e755493fad2cc69d4fe6f280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.197041 1161 ts_tablet_manager.cc:1918] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.197248 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 60edc8fa184542a0b54b89736831730d (DEFAULT_TABLE table=default.Table_cc8b946dd3564d4b95fa7174c5c9bd96 [id=f28fa7e6c39c478c9b313cb0b0963fd5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:29.197965 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78a1bfd87f084d50bd2b602b89c18a2f consensus state has no leader
I20250625 07:52:29.198006 1163 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.198952 1163 ts_tablet_manager.cc:1397] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250625 07:52:29.199666 1163 raft_consensus.cc:357] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.199812 1163 raft_consensus.cc:383] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.199852 1163 raft_consensus.cc:738] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.199827 596 catalog_manager.cc:5582] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:29.200053 1163 consensus_queue.cc:260] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.200222 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 02ec8c9e6340497abe802c7345f42921 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]). Expected version 5 got 0
I20250625 07:52:29.200471 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:29.200736 1163 ts_tablet_manager.cc:1428] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:29.202240 757 tablet_service.cc:1468] Processing CreateTablet for tablet d63d3c33d46c4a7ea59c366416da1d43 (DEFAULT_TABLE table=default.Table_e0146bda63784b9b8882071ad0c5c931 [id=30f80d5e87ec4f10ae072aa51e704e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.204602 1163 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.206230 1163 tablet_bootstrap.cc:654] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.206517 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6606a9559e6b47d28669083af5aee5bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.206919 1162 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.207127 1162 ts_tablet_manager.cc:1397] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20250625 07:52:29.207428 1160 ts_tablet_manager.cc:1918] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.207533 1160 log.cc:1199] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3b9491eb269948a29888a7301ea75d42
I20250625 07:52:29.207844 1160 ts_tablet_manager.cc:1939] T 3b9491eb269948a29888a7301ea75d42 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.208242 1162 raft_consensus.cc:357] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.208482 1162 raft_consensus.cc:383] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.208590 1162 raft_consensus.cc:738] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.208886 1162 consensus_queue.cc:260] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.209417 1162 ts_tablet_manager.cc:1428] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:29.209471 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_444c92729289455eaf1a979360c97e5e" } modify_external_catalogs: true
I20250625 07:52:29.209682 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_444c92729289455eaf1a979360c97e5e" } modify_external_catalogs: true
I20250625 07:52:29.211733 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 1c1a1685d87e45e1857af0b3532e7b10 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.213204 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60edc8fa184542a0b54b89736831730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.218592 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499473d85b284d6cb773148ab276992e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.220718 1158 tablet_bootstrap.cc:654] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.220916 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9f10a26894418795f1c928292afd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.227514 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4709f33f040149c49bd273524c9fd4f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.228973 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d3c33d46c4a7ea59c366416da1d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.230515 1162 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.233923 881 tablet_service.cc:1468] Processing CreateTablet for tablet 60edc8fa184542a0b54b89736831730d (DEFAULT_TABLE table=default.Table_cc8b946dd3564d4b95fa7174c5c9bd96 [id=f28fa7e6c39c478c9b313cb0b0963fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.234411 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60edc8fa184542a0b54b89736831730d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.236768 1162 tablet_bootstrap.cc:654] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.236924 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 901) took 1850 ms. Trace:
I20250625 07:52:29.237118 1160 rpcz_store.cc:276] 0625 07:52:27.385994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.386423 (+ 429us) service_pool.cc:225] Handling call
0625 07:52:28.856528 (+1470105us) ts_tablet_manager.cc:1145] Deleting tablet 3b9491eb269948a29888a7301ea75d42
0625 07:52:28.856536 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:28.875434 (+ 18898us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.207328 (+331894us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.236901 (+ 29573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233109}
I20250625 07:52:29.237663 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 917) took 1589 ms. Trace:
I20250625 07:52:29.237809 1160 rpcz_store.cc:276] 0625 07:52:27.648430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.648501 (+ 71us) service_pool.cc:225] Handling call
0625 07:52:29.237543 (+1589042us) ts_tablet_manager.cc:1145] Deleting tablet 4f1c150fb54c405c8dc61ac133d98b13
0625 07:52:29.237553 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.237645 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586500}
I20250625 07:52:29.238219 1160 tablet_replica.cc:331] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.238430 1160 raft_consensus.cc:2241] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.238621 1160 raft_consensus.cc:2270] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.239224 1160 ts_tablet_manager.cc:1905] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.242415 882 tablet_service.cc:1468] Processing CreateTablet for tablet 494451c0c2514b08ae6be20391a7bcf8 (DEFAULT_TABLE table=default.Table_3245e577c1b349f0b11133c6d1363526 [id=a4f3a18fd7604ad882dea15863c31b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.242975 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494451c0c2514b08ae6be20391a7bcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.244030 1163 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.244177 1163 ts_tablet_manager.cc:1397] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250625 07:52:29.245121 1163 raft_consensus.cc:357] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.245287 1163 raft_consensus.cc:383] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.245347 1163 raft_consensus.cc:738] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.245568 1163 consensus_queue.cc:260] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.248997 1163 ts_tablet_manager.cc:1428] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:29.249122 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 494451c0c2514b08ae6be20391a7bcf8 (DEFAULT_TABLE table=default.Table_3245e577c1b349f0b11133c6d1363526 [id=a4f3a18fd7604ad882dea15863c31b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.249521 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494451c0c2514b08ae6be20391a7bcf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.252122 1160 ts_tablet_manager.cc:1918] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.253144 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c84eab61f1254a67a528c314cd9ab899 (DEFAULT_TABLE table=default.Table_772868a19c60415e8d8d18220e78ab27 [id=7dbeece354644dc896f3c748a5bedd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.253587 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84eab61f1254a67a528c314cd9ab899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.197254 1161 log.cc:1199] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7d9d66546d7c42949dfe5db77f1057fe
I20250625 07:52:29.258291 1161 ts_tablet_manager.cc:1939] T 7d9d66546d7c42949dfe5db77f1057fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.258729 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 880) took 1546 ms. Trace:
I20250625 07:52:29.258873 1161 rpcz_store.cc:276] 0625 07:52:27.712147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.713833 (+ 1686us) service_pool.cc:225] Handling call
0625 07:52:29.094521 (+1380688us) ts_tablet_manager.cc:1145] Deleting tablet 7d9d66546d7c42949dfe5db77f1057fe
0625 07:52:29.094533 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.142707 (+ 48174us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.196934 (+ 54227us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.258714 (+ 61780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380507}
I20250625 07:52:29.259410 1161 tablet_replica.cc:331] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.259634 1161 raft_consensus.cc:2241] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.259838 1161 raft_consensus.cc:2270] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.260418 1161 ts_tablet_manager.cc:1905] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.260653 1160 log.cc:1199] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:29.262557 1160 ts_tablet_manager.cc:1939] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.263387 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 916) took 1642 ms. Trace:
I20250625 07:52:29.263515 1160 rpcz_store.cc:276] 0625 07:52:27.620547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.629538 (+ 8991us) service_pool.cc:225] Handling call
0625 07:52:29.238171 (+1608633us) ts_tablet_manager.cc:1145] Deleting tablet 3ab38e0cdcd14f209e6877f2e4a46e98
0625 07:52:29.238184 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.245010 (+ 6826us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.252023 (+ 7013us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.263368 (+ 11345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587088}
I20250625 07:52:29.264648 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 919) took 1360 ms. Trace:
I20250625 07:52:29.264815 1160 rpcz_store.cc:276] 0625 07:52:27.904622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.904680 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:29.264528 (+1359848us) ts_tablet_manager.cc:1145] Deleting tablet 6d1017546c484c9a89ec039b91c63ad4
0625 07:52:29.264538 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.264632 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359622}
I20250625 07:52:29.265309 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 920) took 1359 ms. Trace:
I20250625 07:52:29.267525 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 (DEFAULT_TABLE table=default.Table_ce15eace6bc5411d808b723c797ed21e [id=8d4a8846aa4f4826880a1ed7fe06ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.266136 885 tablet_service.cc:1468] Processing CreateTablet for tablet f2149b6b3e9743b9a392171626d5b4bb (DEFAULT_TABLE table=default.Table_66a34579b0ea4afca2c08484ccee0b1f [id=2028b2ce2ce0455b8e536b5c2140f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.267729 1160 rpcz_store.cc:276] 0625 07:52:27.905661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.905716 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:29.265201 (+1359485us) ts_tablet_manager.cc:1145] Deleting tablet 32a2929bad2340db9e6e0b5364ec8567
0625 07:52:29.265210 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.265295 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359335}
I20250625 07:52:29.268050 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2149b6b3e9743b9a392171626d5b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:29.269451 713 consensus_peers.cc:487] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:29.268184 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 921) took 1361 ms. Trace:
I20250625 07:52:29.270371 1160 rpcz_store.cc:276] 0625 07:52:27.906409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.906452 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:29.268091 (+1361639us) ts_tablet_manager.cc:1145] Deleting tablet 0bfcef152c9844a59989c532bd60c07d
0625 07:52:29.268100 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.268169 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361476}
I20250625 07:52:29.270869 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 922) took 1351 ms. Trace:
I20250625 07:52:29.270982 1160 rpcz_store.cc:276] 0625 07:52:27.919671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.920937 (+ 1266us) service_pool.cc:225] Handling call
0625 07:52:29.270773 (+1349836us) ts_tablet_manager.cc:1145] Deleting tablet 81451d3745f24cbd9a56307c557e5ec1
0625 07:52:29.270783 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.270855 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349602}
I20250625 07:52:29.271363 1160 tablet_replica.cc:331] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.271569 1160 raft_consensus.cc:2241] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.271744 1160 raft_consensus.cc:2270] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.272293 1160 ts_tablet_manager.cc:1905] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.266939 887 tablet_service.cc:1468] Processing CreateTablet for tablet 95545f3569ad49828ae49136f122e690 (DEFAULT_TABLE table=default.Table_f6785410b52a4eb7a08b256bae5d35e0 [id=c51fd8476a0e43ff86fa68acf94a86fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.273089 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95545f3569ad49828ae49136f122e690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.280944 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656630686bfb49a59c5e7fc8a5cfb4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.285066 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 95545f3569ad49828ae49136f122e690 (DEFAULT_TABLE table=default.Table_f6785410b52a4eb7a08b256bae5d35e0 [id=c51fd8476a0e43ff86fa68acf94a86fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.285486 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95545f3569ad49828ae49136f122e690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.289440 1160 ts_tablet_manager.cc:1918] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.289631 1160 log.cc:1199] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:29.291018 1162 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.291067 1160 ts_tablet_manager.cc:1939] T 7e60d4f9aed44082b9905fbb44ffb110 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.291237 1162 ts_tablet_manager.cc:1397] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.005s
I20250625 07:52:29.291409 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f2149b6b3e9743b9a392171626d5b4bb (DEFAULT_TABLE table=default.Table_66a34579b0ea4afca2c08484ccee0b1f [id=2028b2ce2ce0455b8e536b5c2140f9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.291821 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2149b6b3e9743b9a392171626d5b4bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.291880 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 924) took 1360 ms. Trace:
I20250625 07:52:29.291975 1160 rpcz_store.cc:276] 0625 07:52:27.931112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.931169 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:29.271324 (+1340155us) ts_tablet_manager.cc:1145] Deleting tablet 7e60d4f9aed44082b9905fbb44ffb110
0625 07:52:29.271332 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.275742 (+ 4410us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.283125 (+ 7383us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.291861 (+ 8736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322636}
I20250625 07:52:29.292129 1162 raft_consensus.cc:357] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.292309 1162 raft_consensus.cc:383] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.292368 1162 raft_consensus.cc:738] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.292469 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 925) took 1357 ms. Trace:
I20250625 07:52:29.292583 1162 consensus_queue.cc:260] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.293107 1162 ts_tablet_manager.cc:1428] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:29.293493 1158 tablet_bootstrap.cc:492] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.293614 1158 ts_tablet_manager.cc:1397] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.000s
I20250625 07:52:29.294118 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:29.292641 1160 rpcz_store.cc:276] 0625 07:52:27.934572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.934746 (+ 174us) service_pool.cc:225] Handling call
0625 07:52:29.292359 (+1357613us) ts_tablet_manager.cc:1145] Deleting tablet 6d1017546c484c9a89ec039b91c63ad4
0625 07:52:29.292368 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.292455 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1343569}
I20250625 07:52:29.294494 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 926) took 1323 ms. Trace:
I20250625 07:52:29.294472 1158 raft_consensus.cc:357] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.294657 1158 raft_consensus.cc:383] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.294725 1158 raft_consensus.cc:738] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.294615 1160 rpcz_store.cc:276] 0625 07:52:27.971141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.971196 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:29.294387 (+1323191us) ts_tablet_manager.cc:1145] Deleting tablet b48bc90f80f14bc2b35ce85acafa9028
0625 07:52:29.294397 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.294480 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322989}
I20250625 07:52:29.294955 1158 consensus_queue.cc:260] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.295512 1158 ts_tablet_manager.cc:1428] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:29.295810 1158 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.296932 1162 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.297004 881 tablet_service.cc:1468] Processing CreateTablet for tablet c84eab61f1254a67a528c314cd9ab899 (DEFAULT_TABLE table=default.Table_772868a19c60415e8d8d18220e78ab27 [id=7dbeece354644dc896f3c748a5bedd4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.297338 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84eab61f1254a67a528c314cd9ab899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.299294 1162 tablet_bootstrap.cc:654] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.302332 882 tablet_service.cc:1468] Processing CreateTablet for tablet 652e07d46af74945bf504f7d9015a446 (DEFAULT_TABLE table=default.Table_00fc5f49db204824b892b2ae69796dca [id=5ff5c3c148ad47fbb1121e7c02605e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.302750 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652e07d46af74945bf504f7d9015a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.307720 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 683954dc7ecc4185847f46a05fd2c53d (DEFAULT_TABLE table=default.Table_b527d528908b40de9346966341c2f3fb [id=d091d89c64f840278a5634306587411d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.308131 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683954dc7ecc4185847f46a05fd2c53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.308075 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.308396 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.312266 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.313122 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b98c1f91e4914fb29c6b413e92cb1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.313707 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c9aca996884ffeab88381dd3cb189b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.314023 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 958) took 1035 ms. Trace:
I20250625 07:52:29.314167 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c9aca996884ffeab88381dd3cb189b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.314627 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.314186 1160 rpcz_store.cc:276] 0625 07:52:28.278656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.278974 (+ 318us) service_pool.cc:225] Handling call
0625 07:52:29.313552 (+1034578us) ts_tablet_manager.cc:1145] Deleting tablet b7c5c675d0a7470e95a30a499ad15714
0625 07:52:29.313564 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.314004 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1033647}
I20250625 07:52:29.315101 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.315301 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 954) took 1037 ms. Trace:
I20250625 07:52:29.315414 1160 rpcz_store.cc:276] 0625 07:52:28.278294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.279076 (+ 782us) service_pool.cc:225] Handling call
0625 07:52:29.315200 (+1036124us) ts_tablet_manager.cc:1145] Deleting tablet a3e6a2f10bc14896b6bc01bdc0da3630
0625 07:52:29.315210 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.315289 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036018}
I20250625 07:52:29.315654 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 64a5ff005b424ae5ae9831263abf58ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.315743 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 957) took 1037 ms. Trace:
I20250625 07:52:29.315853 1160 rpcz_store.cc:276] 0625 07:52:28.278575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.279902 (+ 1327us) service_pool.cc:225] Handling call
0625 07:52:29.315665 (+1035763us) ts_tablet_manager.cc:1145] Deleting tablet f5ab0400081848dd872a429685e3b8e2
0625 07:52:29.315673 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.315733 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035635}
I20250625 07:52:29.316200 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 956) took 1037 ms. Trace:
I20250625 07:52:29.316198 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 64a5ff005b424ae5ae9831263abf58ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.316323 1160 rpcz_store.cc:276] 0625 07:52:28.278494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.279861 (+ 1367us) service_pool.cc:225] Handling call
0625 07:52:29.316106 (+1036245us) ts_tablet_manager.cc:1145] Deleting tablet 78f0921af91140c6989de2306326c28f
0625 07:52:29.316114 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.316188 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017882}
I20250625 07:52:29.316671 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 955) took 1038 ms. Trace:
I20250625 07:52:29.316807 1160 rpcz_store.cc:276] 0625 07:52:28.278401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.278943 (+ 542us) service_pool.cc:225] Handling call
0625 07:52:29.316588 (+1037645us) ts_tablet_manager.cc:1145] Deleting tablet f1983df4902f48d2b408a99b440368e7
0625 07:52:29.316596 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.316659 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018222}
I20250625 07:52:29.317231 1160 tablet_replica.cc:331] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.317463 1160 raft_consensus.cc:2241] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.317554 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 652e07d46af74945bf504f7d9015a446 (DEFAULT_TABLE table=default.Table_00fc5f49db204824b892b2ae69796dca [id=5ff5c3c148ad47fbb1121e7c02605e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.317662 1160 raft_consensus.cc:2270] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.317967 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652e07d46af74945bf504f7d9015a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.318354 1160 ts_tablet_manager.cc:1905] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.321079 885 tablet_service.cc:1468] Processing CreateTablet for tablet 683954dc7ecc4185847f46a05fd2c53d (DEFAULT_TABLE table=default.Table_b527d528908b40de9346966341c2f3fb [id=d091d89c64f840278a5634306587411d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.321635 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683954dc7ecc4185847f46a05fd2c53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.325019 887 tablet_service.cc:1468] Processing CreateTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.325472 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.325660 1162 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.325845 1162 ts_tablet_manager.cc:1397] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20250625 07:52:29.326829 1162 raft_consensus.cc:357] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.327055 1162 raft_consensus.cc:383] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.327159 1162 raft_consensus.cc:738] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.327441 1162 consensus_queue.cc:260] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.327987 1162 ts_tablet_manager.cc:1428] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:29.328944 1162 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.336627 1162 tablet_bootstrap.cc:654] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.339349 877 tablet_service.cc:1468] Processing CreateTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 (DEFAULT_TABLE table=default.Table_ce15eace6bc5411d808b723c797ed21e [id=8d4a8846aa4f4826880a1ed7fe06ac10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.340075 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656630686bfb49a59c5e7fc8a5cfb4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.325114 1160 ts_tablet_manager.cc:1918] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.363000 1160 log.cc:1199] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:29.363459 1160 ts_tablet_manager.cc:1939] T cd728ffc2a274deea3fad533d1eaa6da P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.370368 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b8b158d4a1bd40e3ba6c1f74038a5291" } new_table_name: "default.Table_1a133d5654a94e859b2a686a0d5d3a4f" modify_external_catalogs: true
I20250625 07:52:29.371091 1158 tablet_bootstrap.cc:654] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.371922 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_91ecb03f4d35473da357098c9dd96216" } modify_external_catalogs: true
I20250625 07:52:29.372659 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_91ecb03f4d35473da357098c9dd96216" } modify_external_catalogs: true
I20250625 07:52:29.373122 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 1b9f10a26894418795f1c928292afd3d (DEFAULT_TABLE table=default.Table_b80453878cf143caba5ec40c14087313 [id=a3c34c18e755493fad2cc69d4fe6f280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.373345 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 499473d85b284d6cb773148ab276992e (DEFAULT_TABLE table=default.Table_34b97f57ddc44127b6e68fb8a5542a1c [id=07382324ce9f4c31bbc01fb29c076258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.373608 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9f10a26894418795f1c928292afd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.373808 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499473d85b284d6cb773148ab276992e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.374030 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1afe90251c964e369d05e82c7db19326
I20250625 07:52:29.374655 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9c2f9426a94b42f9bbaf640f8b5917cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.375197 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c2f9426a94b42f9bbaf640f8b5917cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.375778 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cded78f0ce7a4af696a4dcaf7b4f5051
I20250625 07:52:29.376319 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e78efc5fea1c4b7ba3aad6f7267bfbfc" } new_table_name: "default.Table_dc62a435b6d44031bc3e5a60a3914e8f" modify_external_catalogs: true
I20250625 07:52:29.376812 879 tablet_service.cc:1468] Processing CreateTablet for tablet 1b9f10a26894418795f1c928292afd3d (DEFAULT_TABLE table=default.Table_b80453878cf143caba5ec40c14087313 [id=a3c34c18e755493fad2cc69d4fe6f280]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.377278 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9f10a26894418795f1c928292afd3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.377466 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 05c040937f7646f5a0fbb98d533250b8
I20250625 07:52:29.378083 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_93934279d504441390d7f1234efdab8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.378535 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93934279d504441390d7f1234efdab8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.379895 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_033160d4bb6d460093d51e5969e61194"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.380306 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_033160d4bb6d460093d51e5969e61194 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.381089 590 master_service.cc:948] ReplaceTablet: received request to replace tablet f1acadde98f54b65830c57df55b06ec3 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.381067 893 tablet_service.cc:1468] Processing CreateTablet for tablet 499473d85b284d6cb773148ab276992e (DEFAULT_TABLE table=default.Table_34b97f57ddc44127b6e68fb8a5542a1c [id=07382324ce9f4c31bbc01fb29c076258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.381467 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499473d85b284d6cb773148ab276992e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.388028 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.389001 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet be7b755d95b14ecea7a691003455f381 (table default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]) successfully deleted
I20250625 07:52:29.389279 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1898ff5b7626478bb997f7a433477498 (table default.Table_8d5544c1432c4978ada968f7fcbfab59 [id=b0330ed5b54243c983112b2afaf66f1f]) successfully deleted
I20250625 07:52:29.389611 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet f191436f3b9244eaa12d211b6f125175 (table default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]) successfully deleted
W20250625 07:52:29.389886 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4167282133f94bfb9870bd08fc6594b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4167282133f94bfb9870bd08fc6594b4 already in progress: creating tablet
W20250625 07:52:29.390117 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f04016cda7f746d49e30be3b1c9c18b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f04016cda7f746d49e30be3b1c9c18b6 already in progress: creating tablet
W20250625 07:52:29.390966 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5b0d357cefb84a239d5be73857b9097b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0d357cefb84a239d5be73857b9097b already in progress: creating tablet
I20250625 07:52:29.391222 753 tablet_service.cc:1515] Processing DeleteTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:29.391270 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 52c1f7cdd88d423788d7d30f15318786 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c1f7cdd88d423788d7d30f15318786 already in progress: creating tablet
W20250625 07:52:29.391515 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ccd929ffd026433c9abd3851bb8e1004 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccd929ffd026433c9abd3851bb8e1004 already in progress: creating tablet
I20250625 07:52:29.391721 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7480c07af7ce4a3889bd8d42d7419d57 (table default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]) successfully deleted
I20250625 07:52:29.391758 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 05c040937f7646f5a0fbb98d533250b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:29.391953 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a396bde4b5964697ad0090eab777ba0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a396bde4b5964697ad0090eab777ba0e already in progress: creating tablet
W20250625 07:52:29.392171 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 2f5fec0e5f364ca093879a8ad2e0bcfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f5fec0e5f364ca093879a8ad2e0bcfc already in progress: creating tablet
I20250625 07:52:29.391222 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1afe90251c964e369d05e82c7db19326 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:29.393893 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet cded78f0ce7a4af696a4dcaf7b4f5051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cded78f0ce7a4af696a4dcaf7b4f5051 already in progress: creating tablet
W20250625 07:52:29.394143 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 05c040937f7646f5a0fbb98d533250b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05c040937f7646f5a0fbb98d533250b8 already in progress: creating tablet
W20250625 07:52:29.394893 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1afe90251c964e369d05e82c7db19326 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1afe90251c964e369d05e82c7db19326 already in progress: creating tablet
I20250625 07:52:29.400729 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet edda0c3f60f64ed0b965eca52c5b3591 (table default.Table_6f4572b322a446d594b17b6e16839583 [id=5d0b8b49ed7f45aeae69c23c09582b34]) successfully deleted
W20250625 07:52:29.406512 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 231ced6803944e24829caf7853da017d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 231ced6803944e24829caf7853da017d already in progress: creating tablet
W20250625 07:52:29.406940 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5de0c5bf60b24808af124c95833d9c25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5de0c5bf60b24808af124c95833d9c25 already in progress: creating tablet
I20250625 07:52:29.408273 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3b9491eb269948a29888a7301ea75d42 (table default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]) successfully deleted
W20250625 07:52:29.408551 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4f1c150fb54c405c8dc61ac133d98b13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1c150fb54c405c8dc61ac133d98b13 already in progress: creating tablet
I20250625 07:52:29.410176 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (table default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]) successfully deleted
W20250625 07:52:29.410501 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d1017546c484c9a89ec039b91c63ad4 already in progress: creating tablet
W20250625 07:52:29.410801 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 32a2929bad2340db9e6e0b5364ec8567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a2929bad2340db9e6e0b5364ec8567 already in progress: creating tablet
W20250625 07:52:29.411108 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0bfcef152c9844a59989c532bd60c07d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bfcef152c9844a59989c532bd60c07d already in progress: creating tablet
W20250625 07:52:29.411383 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 81451d3745f24cbd9a56307c557e5ec1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81451d3745f24cbd9a56307c557e5ec1 already in progress: creating tablet
I20250625 07:52:29.411849 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7e60d4f9aed44082b9905fbb44ffb110 (table default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]) successfully deleted
W20250625 07:52:29.412160 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d1017546c484c9a89ec039b91c63ad4 already in progress: creating tablet
W20250625 07:52:29.412508 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b48bc90f80f14bc2b35ce85acafa9028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b48bc90f80f14bc2b35ce85acafa9028 already in progress: creating tablet
W20250625 07:52:29.412925 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b7c5c675d0a7470e95a30a499ad15714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c5c675d0a7470e95a30a499ad15714 already in progress: creating tablet
W20250625 07:52:29.413270 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a3e6a2f10bc14896b6bc01bdc0da3630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e6a2f10bc14896b6bc01bdc0da3630 already in progress: creating tablet
W20250625 07:52:29.413494 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f5ab0400081848dd872a429685e3b8e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ab0400081848dd872a429685e3b8e2 already in progress: creating tablet
W20250625 07:52:29.413753 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 78f0921af91140c6989de2306326c28f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78f0921af91140c6989de2306326c28f already in progress: creating tablet
W20250625 07:52:29.416392 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f1983df4902f48d2b408a99b440368e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1983df4902f48d2b408a99b440368e7 already in progress: creating tablet
I20250625 07:52:29.418251 876 tablet_service.cc:1468] Processing CreateTablet for tablet d63d3c33d46c4a7ea59c366416da1d43 (DEFAULT_TABLE table=default.Table_e0146bda63784b9b8882071ad0c5c931 [id=30f80d5e87ec4f10ae072aa51e704e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.418498 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d63d3c33d46c4a7ea59c366416da1d43 (DEFAULT_TABLE table=default.Table_e0146bda63784b9b8882071ad0c5c931 [id=30f80d5e87ec4f10ae072aa51e704e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.418778 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 1afe90251c964e369d05e82c7db19326 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.419315 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 1afe90251c964e369d05e82c7db19326 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.419540 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d3c33d46c4a7ea59c366416da1d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.419647 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.419981 880 tablet_service.cc:1515] Processing DeleteTablet for tablet cded78f0ce7a4af696a4dcaf7b4f5051 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.420456 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 05c040937f7646f5a0fbb98d533250b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.421196 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7d9d66546d7c42949dfe5db77f1057fe (table default.Table_3b26428e16b14c4aaaf95f0f92b3670c [id=60fd5cf37619424381543d3cdc94167d]) successfully deleted
I20250625 07:52:29.418892 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d63d3c33d46c4a7ea59c366416da1d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.423411 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 05c040937f7646f5a0fbb98d533250b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.448797 1850 raft_consensus.cc:491] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.448971 1850 raft_consensus.cc:513] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.449541 1850 leader_election.cc:290] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.449796 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet da895c90d6ee40379fe8181c2dbd7a1e
I20250625 07:52:29.450680 880 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.450845 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:29.451262 713 leader_election.cc:343] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2771159858fc41a1ad930229986edd92
I20250625 07:52:29.452085 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a9945f38ef148caac59acaadcb2ef4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.452940 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1c1a1685d87e45e1857af0b3532e7b10
I20250625 07:52:29.453624 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a354a11c894b8f992f3d3cdd66cc06) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.454707 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 37fe97b46b874771ae5ca8c849720118 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.456028 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2771159858fc41a1ad930229986edd92" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:29.456627 713 leader_election.cc:343] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.456775 713 leader_election.cc:304] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.457046 1850 raft_consensus.cc:2747] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.460316 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet c65f257ae653488a8b49b0b523d2cf82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.460557 755 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.460901 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a354a11c894b8f992f3d3cdd66cc06) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.461076 753 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.461570 1166 tablet_replica.cc:331] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.461786 1166 raft_consensus.cc:2241] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.461963 1166 raft_consensus.cc:2270] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.462427 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.462926 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:29.463006 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.463207 591 catalog_manager.cc:6652] ReplaceTablet: tablet 1c1a1685d87e45e1857af0b3532e7b10 deleted and replaced by tablet a1a354a11c894b8f992f3d3cdd66cc06
I20250625 07:52:29.463666 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c1a1685d87e45e1857af0b3532e7b10 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a354a11c894b8f992f3d3cdd66cc06) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.464556 1166 ts_tablet_manager.cc:1905] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.465946 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1bf45c7de2ca4492bbff31a7ecd9404a" } new_table_name: "default.Table_97bafcae580545d58042a9e2bee40c93" modify_external_catalogs: true
W20250625 07:52:29.467885 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet da895c90d6ee40379fe8181c2dbd7a1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da895c90d6ee40379fe8181c2dbd7a1e already in progress: creating tablet
I20250625 07:52:29.467976 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 964) took 1044 ms. Trace:
W20250625 07:52:29.468149 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1c1a1685d87e45e1857af0b3532e7b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1a1685d87e45e1857af0b3532e7b10 already in progress: creating tablet
I20250625 07:52:29.468087 1160 rpcz_store.cc:276] 0625 07:52:28.423781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.423849 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:29.317185 (+893336us) ts_tablet_manager.cc:1145] Deleting tablet cd728ffc2a274deea3fad533d1eaa6da
0625 07:52:29.317195 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.321526 (+ 4331us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.325004 (+ 3478us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.467945 (+142941us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":893129}
I20250625 07:52:29.468544 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet cd728ffc2a274deea3fad533d1eaa6da (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:29.468431 597 catalog_manager.cc:5582] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:29.469631 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.470046 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.470261 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:29.470600 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1a653e84823d467f9beb897f2bfb3b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a653e84823d467f9beb897f2bfb3b3b already in progress: creating tablet
I20250625 07:52:29.470623 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
W20250625 07:52:29.470886 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 381ae5ef5df9437d9bfe0a51987dfe62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 381ae5ef5df9437d9bfe0a51987dfe62 already in progress: creating tablet
I20250625 07:52:29.470922 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:29.471097 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:29.471177 1160 tablet_replica.cc:331] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:29.471259 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4243f6f8c1154fc89b3aa1874441b98d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f6f8c1154fc89b3aa1874441b98d already in progress: creating tablet
I20250625 07:52:29.471422 1160 raft_consensus.cc:2241] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:29.471463 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 51744ae4b84e47cea49fb54023777a41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51744ae4b84e47cea49fb54023777a41 already in progress: creating tablet
I20250625 07:52:29.471593 1160 raft_consensus.cc:2270] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:29.471686 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fdad7dc1b27e4d87a747642a6d47230c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdad7dc1b27e4d87a747642a6d47230c already in progress: creating tablet
W20250625 07:52:29.471905 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b98c1f91e4914fb29c6b413e92cb1929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98c1f91e4914fb29c6b413e92cb1929 already in progress: creating tablet
W20250625 07:52:29.472134 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 47c9aca996884ffeab88381dd3cb189b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c9aca996884ffeab88381dd3cb189b already in progress: creating tablet
I20250625 07:52:29.472219 1160 ts_tablet_manager.cc:1905] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:29.472334 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a80d81cc2d524f318b858a9bd97a89b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80d81cc2d524f318b858a9bd97a89b8 already in progress: creating tablet
W20250625 07:52:29.472565 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 64a5ff005b424ae5ae9831263abf58ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a5ff005b424ae5ae9831263abf58ef already in progress: creating tablet
I20250625 07:52:29.475481 1166 ts_tablet_manager.cc:1918] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.475649 1166 log.cc:1199] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:29.476152 1166 ts_tablet_manager.cc:1939] T ba0735d576f947d395bee8d1b865f2fd P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:29.477481 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet ba0735d576f947d395bee8d1b865f2fd (table default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]) successfully deleted
I20250625 07:52:29.477802 1166 tablet_replica.cc:331] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.478066 1989 log.cc:826] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Log is configured to *not* fsync() on all Append() calls
I20250625 07:52:29.478438 1166 raft_consensus.cc:2241] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.479069 1166 raft_consensus.cc:2270] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.480036 1166 ts_tablet_manager.cc:1905] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.480680 1158 tablet_bootstrap.cc:492] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.480840 1158 ts_tablet_manager.cc:1397] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.185s user 0.003s sys 0.000s
I20250625 07:52:29.481534 1158 raft_consensus.cc:357] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.481705 1158 raft_consensus.cc:383] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.481768 1158 raft_consensus.cc:738] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.481997 1158 consensus_queue.cc:260] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.482527 1158 ts_tablet_manager.cc:1428] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:29.482833 1158 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.484417 1158 tablet_bootstrap.cc:654] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.485472 1793 raft_consensus.cc:491] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.485585 1793 raft_consensus.cc:513] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.486214 1793 leader_election.cc:290] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.487010 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.487744 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:29.487951 777 raft_consensus.cc:2466] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
W20250625 07:52:29.488487 850 leader_election.cc:343] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.489662 1161 ts_tablet_manager.cc:1918] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.489754 1161 log.cc:1199] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/81976db0a2f94128a89842a12f067231
I20250625 07:52:29.489956 851 leader_election.cc:304] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.490056 1161 ts_tablet_manager.cc:1939] T 81976db0a2f94128a89842a12f067231 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.490341 1793 raft_consensus.cc:2802] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:29.490396 1793 raft_consensus.cc:491] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:29.490434 1793 raft_consensus.cc:3058] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:29.492213 1163 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.493656 1163 tablet_bootstrap.cc:654] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:29.495738 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d705b102654747b5a9f16ae9290f96b4 consensus state has no leader
I20250625 07:52:29.495777 1162 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.495901 1162 ts_tablet_manager.cc:1397] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.167s user 0.036s sys 0.056s
I20250625 07:52:29.496148 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e22e84e3af7b408ebf73a2c0cdb15964" } new_table_name: "default.Table_9c66e6d99ff14832b91ad7145582e1e5" modify_external_catalogs: true
I20250625 07:52:29.496600 1162 raft_consensus.cc:357] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.496784 1162 raft_consensus.cc:383] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.496829 1162 raft_consensus.cc:738] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.497140 1162 consensus_queue.cc:260] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.499531 1793 raft_consensus.cc:513] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.500555 1793 leader_election.cc:290] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.500818 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:29.501349 850 leader_election.cc:343] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.504802 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a41ffd566593499299aba5513ccb2f30
I20250625 07:52:29.505218 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:29.505388 777 raft_consensus.cc:3058] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:29.506382 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a [id=b0f32fb69f33447db7620299affa88c8]). Expected version 3 got 0
I20250625 07:52:29.507244 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8c064e83460b4a58b1092ef7ae226598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.507536 1160 ts_tablet_manager.cc:1918] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.507644 1160 log.cc:1199] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1afe90251c964e369d05e82c7db19326
W20250625 07:52:29.507892 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c064e83460b4a58b1092ef7ae226598 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.507961 1160 ts_tablet_manager.cc:1939] T 1afe90251c964e369d05e82c7db19326 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.507990 882 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.508440 1162 ts_tablet_manager.cc:1428] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250625 07:52:29.510582 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1afe90251c964e369d05e82c7db19326 (table default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]) successfully deleted
W20250625 07:52:29.510825 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet cded78f0ce7a4af696a4dcaf7b4f5051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cded78f0ce7a4af696a4dcaf7b4f5051 already in progress: creating tablet
W20250625 07:52:29.510984 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 05c040937f7646f5a0fbb98d533250b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05c040937f7646f5a0fbb98d533250b8 already in progress: creating tablet
W20250625 07:52:29.511153 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet da895c90d6ee40379fe8181c2dbd7a1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da895c90d6ee40379fe8181c2dbd7a1e already in progress: creating tablet
W20250625 07:52:29.511293 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1c1a1685d87e45e1857af0b3532e7b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1a1685d87e45e1857af0b3532e7b10 already in progress: creating tablet
W20250625 07:52:29.511477 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a41ffd566593499299aba5513ccb2f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ffd566593499299aba5513ccb2f30 already in progress: creating tablet
I20250625 07:52:29.514612 1162 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.517196 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.517784 1162 tablet_bootstrap.cc:654] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.517894 1163 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.518023 1163 ts_tablet_manager.cc:1397] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250625 07:52:29.518222 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.518623 1163 raft_consensus.cc:357] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.518788 1163 raft_consensus.cc:383] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.518851 1163 raft_consensus.cc:738] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.519174 1163 consensus_queue.cc:260] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.519768 1163 ts_tablet_manager.cc:1428] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:29.520015 1163 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.520462 1976 raft_consensus.cc:491] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.520591 1976 raft_consensus.cc:513] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.521245 1976 leader_election.cc:290] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:29.521319 1163 tablet_bootstrap.cc:654] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:29.522182 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cdbdd1e3b734463ab452bafecb75822 consensus state has no leader
I20250625 07:52:29.523213 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 881) took 1704 ms. Trace:
I20250625 07:52:29.523304 1161 rpcz_store.cc:276] 0625 07:52:27.818543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.818605 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:29.259365 (+1440760us) ts_tablet_manager.cc:1145] Deleting tablet 81976db0a2f94128a89842a12f067231
0625 07:52:29.259376 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.263459 (+ 4083us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.489566 (+226107us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.523191 (+ 33625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1440591}
I20250625 07:52:29.523762 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 882) took 1619 ms. Trace:
I20250625 07:52:29.523845 1161 rpcz_store.cc:276] 0625 07:52:27.904285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.904408 (+ 123us) service_pool.cc:225] Handling call
0625 07:52:29.523632 (+1619224us) ts_tablet_manager.cc:1145] Deleting tablet 6d1017546c484c9a89ec039b91c63ad4
0625 07:52:29.523645 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.523746 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548313}
I20250625 07:52:29.524168 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 883) took 1618 ms. Trace:
I20250625 07:52:29.524093 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f260806f3d3d4af59e7c35222a769938" } new_table_name: "default.Table_feb8b5a319ea4a7a8372a98f6eb47f7e" modify_external_catalogs: true
I20250625 07:52:29.524237 1161 rpcz_store.cc:276] 0625 07:52:27.905330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.905401 (+ 71us) service_pool.cc:225] Handling call
0625 07:52:29.524068 (+1618667us) ts_tablet_manager.cc:1145] Deleting tablet 32a2929bad2340db9e6e0b5364ec8567
0625 07:52:29.524078 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.524153 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548642}
I20250625 07:52:29.524529 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 884) took 1617 ms. Trace:
I20250625 07:52:29.524581 1161 rpcz_store.cc:276] 0625 07:52:27.906891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.906991 (+ 100us) service_pool.cc:225] Handling call
0625 07:52:29.524450 (+1617459us) ts_tablet_manager.cc:1145] Deleting tablet 0bfcef152c9844a59989c532bd60c07d
0625 07:52:29.524459 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.524519 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548942}
I20250625 07:52:29.524901 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 885) took 1605 ms. Trace:
I20250625 07:52:29.524919 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f1acadde98f54b65830c57df55b06ec3
I20250625 07:52:29.524957 1161 rpcz_store.cc:276] 0625 07:52:27.919150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.919212 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:29.524810 (+1605598us) ts_tablet_manager.cc:1145] Deleting tablet 81451d3745f24cbd9a56307c557e5ec1
0625 07:52:29.524818 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.524888 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549244}
W20250625 07:52:29.525035 985 leader_election.cc:343] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:29.525199 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 886) took 1591 ms. Trace:
I20250625 07:52:29.525259 1161 rpcz_store.cc:276] 0625 07:52:27.934197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.934262 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:29.525120 (+1590858us) ts_tablet_manager.cc:1145] Deleting tablet 6d1017546c484c9a89ec039b91c63ad4
0625 07:52:29.525127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.525187 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540338}
I20250625 07:52:29.525439 590 catalog_manager.cc:6652] ReplaceTablet: tablet f1acadde98f54b65830c57df55b06ec3 deleted and replaced by tablet 8a13381cae324e9494a2497d0190a868
I20250625 07:52:29.525521 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 888) took 1554 ms. Trace:
I20250625 07:52:29.525580 1161 rpcz_store.cc:276] 0625 07:52:27.970852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.970926 (+ 74us) service_pool.cc:225] Handling call
0625 07:52:29.525438 (+1554512us) ts_tablet_manager.cc:1145] Deleting tablet b48bc90f80f14bc2b35ce85acafa9028
0625 07:52:29.525446 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.525508 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538394}
I20250625 07:52:29.525789 1161 tablet_replica.cc:331] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:29.525769 986 leader_election.cc:343] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:29.525749 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aef7709d3ff64d91a06278b1080f77ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.525866 986 leader_election.cc:304] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.525977 1161 raft_consensus.cc:2241] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.526118 1161 raft_consensus.cc:2270] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:29.526297 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef7709d3ff64d91a06278b1080f77ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.526587 1161 ts_tablet_manager.cc:1905] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.527212 1976 raft_consensus.cc:2747] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.527670 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 81976db0a2f94128a89842a12f067231 (table default.Table_3ed8b54f47094fea9a6e3de748e5ca57 [id=967112db07f149eb895f89db1f0f85a4]) successfully deleted
I20250625 07:52:29.527853 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dc4bf2e1a50b434f838b277b1edc3471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.527954 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d1017546c484c9a89ec039b91c63ad4 already in progress: creating tablet
W20250625 07:52:29.528177 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 32a2929bad2340db9e6e0b5364ec8567 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32a2929bad2340db9e6e0b5364ec8567 already in progress: creating tablet
W20250625 07:52:29.528404 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc4bf2e1a50b434f838b277b1edc3471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.529259 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet f1acadde98f54b65830c57df55b06ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13381cae324e9494a2497d0190a868) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.534175 1158 tablet_bootstrap.cc:492] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.534302 1158 ts_tablet_manager.cc:1397] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.000s
I20250625 07:52:29.534970 1158 raft_consensus.cc:357] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.535144 1158 raft_consensus.cc:383] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.535199 1158 raft_consensus.cc:738] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.535380 1158 consensus_queue.cc:260] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.536069 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f1acadde98f54b65830c57df55b06ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13381cae324e9494a2497d0190a868) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.536307 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.536780 1976 raft_consensus.cc:491] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.536911 1976 raft_consensus.cc:513] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.537042 1161 ts_tablet_manager.cc:1918] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.537143 1161 log.cc:1199] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e41e53efb6f342d2b1e4d642f7aaad10
I20250625 07:52:29.537499 1976 leader_election.cc:290] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:29.538431 1158 ts_tablet_manager.cc:1428] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:29.538889 1158 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:29.539471 985 leader_election.cc:343] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2a9945f38ef148caac59acaadcb2ef4c
W20250625 07:52:29.528378 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0bfcef152c9844a59989c532bd60c07d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bfcef152c9844a59989c532bd60c07d already in progress: creating tablet
W20250625 07:52:29.540272 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 81451d3745f24cbd9a56307c557e5ec1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81451d3745f24cbd9a56307c557e5ec1 already in progress: creating tablet
I20250625 07:52:29.540477 1158 tablet_bootstrap.cc:654] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.540916 1161 ts_tablet_manager.cc:1939] T e41e53efb6f342d2b1e4d642f7aaad10 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.541723 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 891) took 1543 ms. Trace:
I20250625 07:52:29.541815 1161 rpcz_store.cc:276] 0625 07:52:27.998567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:27.998686 (+ 119us) service_pool.cc:225] Handling call
0625 07:52:29.525751 (+1527065us) ts_tablet_manager.cc:1145] Deleting tablet e41e53efb6f342d2b1e4d642f7aaad10
0625 07:52:29.525758 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.531254 (+ 5496us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.536937 (+ 5683us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.541703 (+ 4766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526880}
I20250625 07:52:29.541967 589 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet f84acd45845c4917b7dc8c5ad5232897 (table default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a [id=b0f32fb69f33447db7620299affa88c8]). Expected version 3 got 0
I20250625 07:52:29.545200 589 master_service.cc:948] ReplaceTablet: received request to replace tablet a8a668fb466840e0a41749a4db9a1554 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.546044 1158 tablet_bootstrap.cc:492] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.546180 1158 ts_tablet_manager.cc:1397] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250625 07:52:29.546139 777 raft_consensus.cc:2466] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:29.546797 1158 raft_consensus.cc:357] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.546949 1158 raft_consensus.cc:383] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.546993 1158 raft_consensus.cc:738] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.547161 1158 consensus_queue.cc:260] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.547560 1158 ts_tablet_manager.cc:1428] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 07:52:29.547806 1158 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.548985 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 917) took 1312 ms. Trace:
I20250625 07:52:29.549077 1161 rpcz_store.cc:276] 0625 07:52:28.236171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.242032 (+ 5861us) service_pool.cc:225] Handling call
0625 07:52:29.548856 (+1306824us) ts_tablet_manager.cc:1145] Deleting tablet a3e6a2f10bc14896b6bc01bdc0da3630
0625 07:52:29.548865 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.548965 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304768}
I20250625 07:52:29.549175 851 leader_election.cc:304] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.549556 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 920) took 1298 ms. Trace:
I20250625 07:52:29.549603 1793 raft_consensus.cc:2802] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:29.549664 1163 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.549629 1161 rpcz_store.cc:276] 0625 07:52:28.251047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.251090 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:29.549449 (+1298359us) ts_tablet_manager.cc:1145] Deleting tablet f5ab0400081848dd872a429685e3b8e2
0625 07:52:29.549460 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.549540 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298201}
I20250625 07:52:29.549772 1163 ts_tablet_manager.cc:1397] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250625 07:52:29.550374 1163 raft_consensus.cc:357] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.550514 1163 raft_consensus.cc:383] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.550560 1163 raft_consensus.cc:738] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:29.550909 986 leader_election.cc:343] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2a9945f38ef148caac59acaadcb2ef4c
I20250625 07:52:29.550967 588 master_service.cc:948] ReplaceTablet: received request to replace tablet c858e286fbba450e81c6cc55f0f2b613 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.551004 986 leader_election.cc:304] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.551016 1163 consensus_queue.cc:260] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.551492 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 921) took 1300 ms. Trace:
I20250625 07:52:29.551575 1161 rpcz_store.cc:276] 0625 07:52:28.251173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.251280 (+ 107us) service_pool.cc:225] Handling call
0625 07:52:29.551373 (+1300093us) ts_tablet_manager.cc:1145] Deleting tablet b7c5c675d0a7470e95a30a499ad15714
0625 07:52:29.551385 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.551475 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299953}
W20250625 07:52:29.540520 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6d1017546c484c9a89ec039b91c63ad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d1017546c484c9a89ec039b91c63ad4 already in progress: creating tablet
W20250625 07:52:29.553088 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b48bc90f80f14bc2b35ce85acafa9028 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b48bc90f80f14bc2b35ce85acafa9028 already in progress: creating tablet
I20250625 07:52:29.553407 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e41e53efb6f342d2b1e4d642f7aaad10 (table default.Table_913332b4f3224e12941ecc4c9d33f0d2 [id=c2c95c5f1b914c829fa93e8955558fa2]) successfully deleted
I20250625 07:52:29.556932 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 918) took 1306 ms. Trace:
I20250625 07:52:29.557036 1161 rpcz_store.cc:276] 0625 07:52:28.250839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.256740 (+ 5901us) service_pool.cc:225] Handling call
0625 07:52:29.556793 (+1300053us) ts_tablet_manager.cc:1145] Deleting tablet f1983df4902f48d2b408a99b440368e7
0625 07:52:29.556804 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.556911 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299904}
I20250625 07:52:29.557389 2028 raft_consensus.cc:2747] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.557799 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 919) took 1306 ms. Trace:
I20250625 07:52:29.557977 1161 rpcz_store.cc:276] 0625 07:52:28.250941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.261177 (+ 10236us) service_pool.cc:225] Handling call
0625 07:52:29.557690 (+1296513us) ts_tablet_manager.cc:1145] Deleting tablet 78f0921af91140c6989de2306326c28f
0625 07:52:29.557700 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.557782 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296325}
I20250625 07:52:29.558495 1161 tablet_replica.cc:331] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.558844 1161 raft_consensus.cc:2241] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.559046 1161 raft_consensus.cc:2270] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.559267 1163 ts_tablet_manager.cc:1428] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250625 07:52:29.559566 1163 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.559782 1161 ts_tablet_manager.cc:1905] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.560199 1793 raft_consensus.cc:695] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:29.560472 1793 consensus_queue.cc:237] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.561312 1163 tablet_bootstrap.cc:654] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:29.561646 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d705b102654747b5a9f16ae9290f96b4 (table default.Table_1a133d5654a94e859b2a686a0d5d3a4f [id=02f47cef001a4202994fad0596557c9a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d705b102654747b5a9f16ae9290f96b4 consensus state has no leader
I20250625 07:52:29.562620 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 37fe97b46b874771ae5ca8c849720118
I20250625 07:52:29.563038 596 catalog_manager.cc:6652] ReplaceTablet: tablet 37fe97b46b874771ae5ca8c849720118 deleted and replaced by tablet 7d06e9aae7eb408c99d97e52fc9f691c
I20250625 07:52:29.563093 882 tablet_service.cc:1515] Processing DeleteTablet for tablet f1acadde98f54b65830c57df55b06ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13381cae324e9494a2497d0190a868) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.565279 1158 tablet_bootstrap.cc:654] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.566115 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 37fe97b46b874771ae5ca8c849720118 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d06e9aae7eb408c99d97e52fc9f691c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.570235 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:29.571990 1850 raft_consensus.cc:491] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.572160 1850 raft_consensus.cc:513] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.572804 1850 leader_election.cc:290] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.573952 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ec29e4294864fbabdff5a04dcf5b163" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:29.574800 713 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5ec29e4294864fbabdff5a04dcf5b163
W20250625 07:52:29.575045 713 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.575129 713 leader_election.cc:304] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.575582 1850 raft_consensus.cc:2747] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:29.577013 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0352225d645c408ba8c74273ee5f56e6 consensus state has no leader
I20250625 07:52:29.577646 1163 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.577780 1163 ts_tablet_manager.cc:1397] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20250625 07:52:29.578565 1163 raft_consensus.cc:357] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.578722 1163 raft_consensus.cc:383] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.578770 1163 raft_consensus.cc:738] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.578966 1163 consensus_queue.cc:260] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.579564 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:29.582610 1158 tablet_bootstrap.cc:492] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.582752 1158 ts_tablet_manager.cc:1397] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250625 07:52:29.583462 1158 raft_consensus.cc:357] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.584008 1158 raft_consensus.cc:383] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.584074 1158 raft_consensus.cc:738] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.584280 1158 consensus_queue.cc:260] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.585091 1163 ts_tablet_manager.cc:1428] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:29.585366 1163 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.588148 1158 ts_tablet_manager.cc:1428] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:29.588423 1158 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.593497 1161 ts_tablet_manager.cc:1918] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.593629 1161 log.cc:1199] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2f63fdf2c8464c8fb58f412263d4e5c5
I20250625 07:52:29.593811 1163 tablet_bootstrap.cc:654] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.593935 1161 ts_tablet_manager.cc:1939] T 2f63fdf2c8464c8fb58f412263d4e5c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.594265 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 926) took 1178 ms. Trace:
I20250625 07:52:29.594350 1161 rpcz_store.cc:276] 0625 07:52:28.416294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:28.416353 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:29.558444 (+1142091us) ts_tablet_manager.cc:1145] Deleting tablet 2f63fdf2c8464c8fb58f412263d4e5c5
0625 07:52:29.558455 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:29.578520 (+ 20065us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.593382 (+ 14862us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:29.594248 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141856}
I20250625 07:52:29.594899 1161 tablet_replica.cc:331] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.595091 1161 raft_consensus.cc:2241] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.595257 1161 raft_consensus.cc:2270] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.595816 1161 ts_tablet_manager.cc:1905] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.596207 1158 tablet_bootstrap.cc:654] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.598304 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a2c86b9769ce4cb695819e5d7601ef14" } modify_external_catalogs: true
I20250625 07:52:29.598464 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a2c86b9769ce4cb695819e5d7601ef14" } modify_external_catalogs: true
I20250625 07:52:29.602605 1166 ts_tablet_manager.cc:1918] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.603842 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8c9a216aabd74a76990345c02842ea74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.604477 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c9a216aabd74a76990345c02842ea74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.604513 1163 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.604635 1163 ts_tablet_manager.cc:1397] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250625 07:52:29.604969 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8813030417384a41a121df21b15656e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.605371 1163 raft_consensus.cc:357] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:29.605497 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8813030417384a41a121df21b15656e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.605531 1163 raft_consensus.cc:383] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.605576 1163 raft_consensus.cc:738] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.605762 1163 consensus_queue.cc:260] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.609562 1793 consensus_queue.cc:1035] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:29.610287 1163 ts_tablet_manager.cc:1428] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250625 07:52:29.610574 1163 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.611433 2028 raft_consensus.cc:491] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.611578 2028 raft_consensus.cc:513] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.612140 2028 leader_election.cc:290] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:29.613284 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 745cd19f3b964febbba91f9bf95755c8 consensus state has no leader
W20250625 07:52:29.613725 986 leader_election.cc:343] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:29.614813 1158 tablet_bootstrap.cc:492] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.614955 1158 ts_tablet_manager.cc:1397] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250625 07:52:29.614961 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cb3c32e8a2f246b88ee85462a6d3ff99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.615468 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb3c32e8a2f246b88ee85462a6d3ff99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.615667 1158 raft_consensus.cc:357] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.615815 1158 raft_consensus.cc:383] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.615878 1158 raft_consensus.cc:738] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:29.616523 985 leader_election.cc:343] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:29.616670 985 leader_election.cc:304] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.617300 2028 raft_consensus.cc:2747] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.618191 1163 tablet_bootstrap.cc:654] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:29.620013 583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 66639.8us
I20250625 07:52:29.620502 1158 consensus_queue.cc:260] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.621645 1158 ts_tablet_manager.cc:1428] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:29.623980 1163 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.624132 1163 ts_tablet_manager.cc:1397] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20250625 07:52:29.625152 1163 raft_consensus.cc:357] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.625376 1163 raft_consensus.cc:383] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.625454 1163 raft_consensus.cc:738] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.625676 1163 consensus_queue.cc:260] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.629290 1163 ts_tablet_manager.cc:1428] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 07:52:29.630447 1163 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.635434 1163 tablet_bootstrap.cc:654] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.644913 1793 raft_consensus.cc:491] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.645087 1793 raft_consensus.cc:513] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.645718 1793 leader_election.cc:290] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.650759 1161 ts_tablet_manager.cc:1918] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.650920 1161 log.cc:1199] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/875fa1b46903452eaed1f2235d63dfdf
I20250625 07:52:29.651288 1161 ts_tablet_manager.cc:1939] T 875fa1b46903452eaed1f2235d63dfdf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.656888 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1baf94a051e485eaf4c0e074e147252" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.657804 1158 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:29.658470 850 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.661284 1158 tablet_bootstrap.cc:654] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.623229 1162 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.649998 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1baf94a051e485eaf4c0e074e147252" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:29.664987 1162 ts_tablet_manager.cc:1397] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.150s user 0.005s sys 0.019s
W20250625 07:52:29.665295 851 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.665441 851 leader_election.cc:304] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.665856 2030 raft_consensus.cc:2747] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.666038 1850 raft_consensus.cc:491] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.665935 1162 raft_consensus.cc:357] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.666137 1162 raft_consensus.cc:383] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.666190 1162 raft_consensus.cc:738] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.666158 1850 raft_consensus.cc:513] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.666340 1162 consensus_queue.cc:260] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.666682 1850 leader_election.cc:290] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:29.666713 1166 log.cc:1199] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:29.666815 1162 ts_tablet_manager.cc:1428] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:29.667096 1166 ts_tablet_manager.cc:1939] T f9a94dc1432d4cbdbf3eecacb1429c23 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:29.667625 1162 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:29.667810 713 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:29.669229 1162 tablet_bootstrap.cc:654] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.673266 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e4bf78218f4b82b4f16f59b1704256" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:29.673559 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a3e6a2f10bc14896b6bc01bdc0da3630 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e6a2f10bc14896b6bc01bdc0da3630 already in progress: creating tablet
W20250625 07:52:29.673805 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f5ab0400081848dd872a429685e3b8e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5ab0400081848dd872a429685e3b8e2 already in progress: creating tablet
I20250625 07:52:29.673921 1163 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.640538 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aeb5c70494240aaaad67f311ec1723b" } modify_external_catalogs: true
W20250625 07:52:29.674008 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b7c5c675d0a7470e95a30a499ad15714 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c5c675d0a7470e95a30a499ad15714 already in progress: creating tablet
I20250625 07:52:29.674068 1163 ts_tablet_manager.cc:1397] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.008s
I20250625 07:52:29.674103 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aeb5c70494240aaaad67f311ec1723b" } modify_external_catalogs: true
I20250625 07:52:29.675068 2028 raft_consensus.cc:491] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.675249 2028 raft_consensus.cc:513] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.675788 912 raft_consensus.cc:1273] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:29.675887 2028 leader_election.cc:290] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:29.674172 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f1983df4902f48d2b408a99b440368e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1983df4902f48d2b408a99b440368e7 already in progress: creating tablet
W20250625 07:52:29.676267 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 78f0921af91140c6989de2306326c28f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78f0921af91140c6989de2306326c28f already in progress: creating tablet
I20250625 07:52:29.676427 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2f63fdf2c8464c8fb58f412263d4e5c5 (table default.Table_5946e2f2580849578b2bda787459faef [id=e441385dc1044575b17f64b06f14c648]) successfully deleted
W20250625 07:52:29.676434 713 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.676532 713 leader_election.cc:304] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:29.676587 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 381ae5ef5df9437d9bfe0a51987dfe62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 381ae5ef5df9437d9bfe0a51987dfe62 already in progress: creating tablet
I20250625 07:52:29.676981 1850 raft_consensus.cc:2747] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.677397 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.677608 879 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.677632 1160 tablet_replica.cc:331] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.677706 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 37fe97b46b874771ae5ca8c849720118 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d06e9aae7eb408c99d97e52fc9f691c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.677765 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.677824 1160 raft_consensus.cc:2241] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.677963 1160 raft_consensus.cc:2270] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.678447 1160 ts_tablet_manager.cc:1905] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.678509 1166 tablet_replica.cc:331] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.679004 1166 raft_consensus.cc:2241] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.679443 1166 raft_consensus.cc:2270] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.680195 1166 ts_tablet_manager.cc:1905] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.680491 1163 raft_consensus.cc:357] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.680567 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 37fe97b46b874771ae5ca8c849720118 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d06e9aae7eb408c99d97e52fc9f691c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.680810 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4519b1fbf34ca8b33655f4d402aafc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.680836 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d8d3300befc4f419070ae84725ee96a with delete_type TABLET_DATA_DELETED (replaced by tablet 3cdbdd1e3b734463ab452bafecb75822) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.680986 1163 raft_consensus.cc:383] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.681069 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:29.681439 1163 raft_consensus.cc:738] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.681663 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet f9a94dc1432d4cbdbf3eecacb1429c23 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
I20250625 07:52:29.681783 1163 consensus_queue.cc:260] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:29.681502 713 consensus_peers.cc:487] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:29.653544 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6f4572b322a446d594b17b6e16839583" } modify_external_catalogs: true
I20250625 07:52:29.682196 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6f4572b322a446d594b17b6e16839583" } modify_external_catalogs: true
I20250625 07:52:29.682348 1982 consensus_queue.cc:1035] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:29.684942 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 0024618293d74f559b04258f66616044 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.685031 1160 ts_tablet_manager.cc:1918] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.685129 1160 log.cc:1199] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:29.685412 1160 ts_tablet_manager.cc:1939] T e289ea5553ac4ff983871ece4fc72ee5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.691038 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f84acd45845c4917b7dc8c5ad5232897" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:29.692039 1161 tablet_replica.cc:331] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.692194 985 leader_election.cc:304] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.692925 1163 ts_tablet_manager.cc:1428] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250625 07:52:29.693197 1163 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.694213 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4801dc1c88174eb1838ebc1ebbd93d6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.694840 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4801dc1c88174eb1838ebc1ebbd93d6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.694829 2028 raft_consensus.cc:2747] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.696578 753 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.697180 1162 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.697363 1162 ts_tablet_manager.cc:1397] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
W20250625 07:52:29.698077 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f1acadde98f54b65830c57df55b06ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1acadde98f54b65830c57df55b06ec3 already in progress: creating tablet
I20250625 07:52:29.698529 1162 raft_consensus.cc:357] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.698697 1162 raft_consensus.cc:383] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.698763 1162 raft_consensus.cc:738] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.699039 1162 consensus_queue.cc:260] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.692237 1161 raft_consensus.cc:2241] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.699496 1162 ts_tablet_manager.cc:1428] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:29.699754 1161 raft_consensus.cc:2270] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.699817 1162 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.700728 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 875fa1b46903452eaed1f2235d63dfdf (table default.Table_35e64151eae847128b84b16f6c0ab3ea [id=b7239cfa06b649ab946d863c06190160]) successfully deleted
I20250625 07:52:29.702586 1160 tablet_replica.cc:331] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.703373 1162 tablet_bootstrap.cc:654] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.703382 1160 raft_consensus.cc:2241] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.704174 1160 raft_consensus.cc:2270] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.700381 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet e7100ef72ff14c24be92c21c6eb8a9de with delete_type TABLET_DATA_DELETED (replaced by tablet 4c0bbd21d4aa41b2b92473bffe7dc05a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.705260 1163 tablet_bootstrap.cc:654] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:29.706566 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fdad7dc1b27e4d87a747642a6d47230c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdad7dc1b27e4d87a747642a6d47230c already in progress: creating tablet
I20250625 07:52:29.707229 1161 ts_tablet_manager.cc:1905] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.707377 1160 ts_tablet_manager.cc:1905] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.710464 1162 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.710659 1162 ts_tablet_manager.cc:1397] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20250625 07:52:29.711499 1162 raft_consensus.cc:357] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.712006 1162 raft_consensus.cc:383] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.712116 1162 raft_consensus.cc:738] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.712390 1162 consensus_queue.cc:260] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.711920 1166 ts_tablet_manager.cc:1918] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.712814 1166 log.cc:1199] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:29.713294 1166 ts_tablet_manager.cc:1939] T 5ec29e4294864fbabdff5a04dcf5b163 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:29.713531 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1a653e84823d467f9beb897f2bfb3b3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a653e84823d467f9beb897f2bfb3b3b already in progress: creating tablet
W20250625 07:52:29.713794 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 059dad944716497bbba17e11fb3ba5be consensus state has no leader
I20250625 07:52:29.713804 1162 ts_tablet_manager.cc:1428] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250625 07:52:29.713812 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4243f6f8c1154fc89b3aa1874441b98d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4243f6f8c1154fc89b3aa1874441b98d already in progress: creating tablet
W20250625 07:52:29.714006 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 51744ae4b84e47cea49fb54023777a41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51744ae4b84e47cea49fb54023777a41 already in progress: creating tablet
I20250625 07:52:29.714211 1166 tablet_replica.cc:331] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.714360 1166 raft_consensus.cc:2241] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.714367 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e289ea5553ac4ff983871ece4fc72ee5 (table default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]) successfully deleted
I20250625 07:52:29.714849 1162 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.716419 1162 tablet_bootstrap.cc:654] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.716739 1166 raft_consensus.cc:2270] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.717314 1166 ts_tablet_manager.cc:1905] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.719485 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 5ec29e4294864fbabdff5a04dcf5b163 (table default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]) successfully deleted
W20250625 07:52:29.721324 850 consensus_peers.cc:487] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:29.725199 2030 raft_consensus.cc:491] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.725342 2030 raft_consensus.cc:513] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.725996 2030 leader_election.cc:290] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.726456 1163 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.726589 1163 ts_tablet_manager.cc:1397] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250625 07:52:29.726809 1161 ts_tablet_manager.cc:1918] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.726938 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.727218 1161 log.cc:1199] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6606a9559e6b47d28669083af5aee5bd
I20250625 07:52:29.727361 1163 raft_consensus.cc:357] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.727514 1163 raft_consensus.cc:383] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:29.727562 850 leader_election.cc:343] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.727564 1163 raft_consensus.cc:738] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.728092 1163 consensus_queue.cc:260] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.728371 1161 ts_tablet_manager.cc:1939] T 6606a9559e6b47d28669083af5aee5bd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.728807 1163 ts_tablet_manager.cc:1428] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:29.729024 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6606a9559e6b47d28669083af5aee5bd (table default.Table_b662e261763d437fafa99ee7cade09e7 [id=1622032cfd9b41c6a1e9133a505da9b0]) successfully deleted
I20250625 07:52:29.729101 1163 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:29.730322 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b98c1f91e4914fb29c6b413e92cb1929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b98c1f91e4914fb29c6b413e92cb1929 already in progress: creating tablet
W20250625 07:52:29.731788 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 47c9aca996884ffeab88381dd3cb189b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c9aca996884ffeab88381dd3cb189b already in progress: creating tablet
W20250625 07:52:29.732609 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a80d81cc2d524f318b858a9bd97a89b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a80d81cc2d524f318b858a9bd97a89b8 already in progress: creating tablet
W20250625 07:52:29.734243 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 64a5ff005b424ae5ae9831263abf58ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a5ff005b424ae5ae9831263abf58ef already in progress: creating tablet
I20250625 07:52:29.734745 1161 tablet_replica.cc:331] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.735592 1161 raft_consensus.cc:2241] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.735961 1161 raft_consensus.cc:2270] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.737710 1161 ts_tablet_manager.cc:1905] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.738372 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a8a668fb466840e0a41749a4db9a1554
I20250625 07:52:29.738909 589 catalog_manager.cc:6652] ReplaceTablet: tablet a8a668fb466840e0a41749a4db9a1554 deleted and replaced by tablet a571983779854eb398b3b0b42c34f83a
I20250625 07:52:29.739078 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a668fb466840e0a41749a4db9a1554 with delete_type TABLET_DATA_DELETED (replaced by tablet a571983779854eb398b3b0b42c34f83a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.739282 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40a9b063009d432680f575cbc3dfad3f" } modify_external_catalogs: true
I20250625 07:52:29.739486 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_40a9b063009d432680f575cbc3dfad3f" } modify_external_catalogs: true
I20250625 07:52:29.740446 876 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a668fb466840e0a41749a4db9a1554 with delete_type TABLET_DATA_DELETED (replaced by tablet a571983779854eb398b3b0b42c34f83a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.741300 753 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a668fb466840e0a41749a4db9a1554 with delete_type TABLET_DATA_DELETED (replaced by tablet a571983779854eb398b3b0b42c34f83a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.744081 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:29.744972 851 leader_election.cc:343] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.745114 851 leader_election.cc:304] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.745633 2030 raft_consensus.cc:2747] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.752884 1163 tablet_bootstrap.cc:654] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.769676 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c858e286fbba450e81c6cc55f0f2b613
I20250625 07:52:29.770208 588 catalog_manager.cc:6652] ReplaceTablet: tablet c858e286fbba450e81c6cc55f0f2b613 deleted and replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87
I20250625 07:52:29.773224 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet c858e286fbba450e81c6cc55f0f2b613 with delete_type TABLET_DATA_DELETED (replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.773288 876 tablet_service.cc:1515] Processing DeleteTablet for tablet c858e286fbba450e81c6cc55f0f2b613 with delete_type TABLET_DATA_DELETED (replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.776015 753 tablet_service.cc:1515] Processing DeleteTablet for tablet c858e286fbba450e81c6cc55f0f2b613 with delete_type TABLET_DATA_DELETED (replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.776239 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.772550 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a564b726340d44f485906f969425fdd5" } new_table_name: "default.Table_5f3b2ea742284d1e902ef62d1ea5fbd0" modify_external_catalogs: true
I20250625 07:52:29.776262 755 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.788892 591 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet cf83600b3e5247b48f530a17b04ff3e7
I20250625 07:52:29.789565 1161 ts_tablet_manager.cc:1918] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.797503 2052 tablet.cc:1722] T 02ec8c9e6340497abe802c7345f42921 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:29.799520 1850 raft_consensus.cc:491] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.799670 1850 raft_consensus.cc:513] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.800279 1850 leader_election.cc:290] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.803396 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f9f4b30d1e6049428f31ff5791dcc3a0" } modify_external_catalogs: true
I20250625 07:52:29.803601 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f9f4b30d1e6049428f31ff5791dcc3a0" } modify_external_catalogs: true
I20250625 07:52:29.805299 1163 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.805431 1163 ts_tablet_manager.cc:1397] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.001s
I20250625 07:52:29.806216 1163 raft_consensus.cc:357] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.806377 1163 raft_consensus.cc:383] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.806435 1163 raft_consensus.cc:738] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.806628 1163 consensus_queue.cc:260] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.807097 1163 ts_tablet_manager.cc:1428] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:29.807366 1163 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.758831 1166 ts_tablet_manager.cc:1918] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.809005 1166 log.cc:1199] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/541c983d64f348539979f74199f79e59
I20250625 07:52:29.809373 1166 ts_tablet_manager.cc:1939] T 541c983d64f348539979f74199f79e59 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:29.793833 1158 tablet_bootstrap.cc:492] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.809600 1158 ts_tablet_manager.cc:1397] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.014s
I20250625 07:52:29.804380 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbdbd9230210455ba615d51f1a98267b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.810367 1166 tablet_replica.cc:331] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.810595 1166 raft_consensus.cc:2241] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.810575 1158 raft_consensus.cc:357] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.810760 1158 raft_consensus.cc:383] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:29.810777 713 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.810830 1158 raft_consensus.cc:738] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:29.810961 713 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:29.811077 713 leader_election.cc:304] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.811064 1158 consensus_queue.cc:260] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.811554 1158 ts_tablet_manager.cc:1428] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:29.811854 1158 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.813158 1161 log.cc:1199] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6463d2bc39034ee7ad1b8bfcfcb4b0fe
I20250625 07:52:29.810961 1166 raft_consensus.cc:2270] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.813527 1161 ts_tablet_manager.cc:1939] T 6463d2bc39034ee7ad1b8bfcfcb4b0fe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.813956 1166 ts_tablet_manager.cc:1905] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.814163 2045 raft_consensus.cc:2747] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.814527 1158 tablet_bootstrap.cc:654] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.814827 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 541c983d64f348539979f74199f79e59 (table default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]) successfully deleted
I20250625 07:52:29.818884 2053 tablet.cc:1722] T 02ec8c9e6340497abe802c7345f42921 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:29.823557 1162 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.823807 1162 ts_tablet_manager.cc:1397] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.004s
I20250625 07:52:29.824651 1162 raft_consensus.cc:357] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.824923 1162 raft_consensus.cc:383] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.825057 1162 raft_consensus.cc:738] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.825354 1162 consensus_queue.cc:260] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.825922 1162 ts_tablet_manager.cc:1428] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:29.826297 1162 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.828008 1162 tablet_bootstrap.cc:654] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.828097 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:29.828085 1160 ts_tablet_manager.cc:1918] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.829161 1160 log.cc:1199] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:29.831158 1160 ts_tablet_manager.cc:1939] T a1baf94a051e485eaf4c0e074e147252 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.832458 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6463d2bc39034ee7ad1b8bfcfcb4b0fe (table default.Table_12188272bb8e4c26bb94db8721d37c26 [id=58cee91e0a014b2ea44ba7a6538e667b]) successfully deleted
W20250625 07:52:29.835256 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1afe90251c964e369d05e82c7db19326 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1afe90251c964e369d05e82c7db19326 already in progress: creating tablet
W20250625 07:52:29.835963 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet cded78f0ce7a4af696a4dcaf7b4f5051 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cded78f0ce7a4af696a4dcaf7b4f5051 already in progress: creating tablet
W20250625 07:52:29.836557 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 05c040937f7646f5a0fbb98d533250b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05c040937f7646f5a0fbb98d533250b8 already in progress: creating tablet
W20250625 07:52:29.837198 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet da895c90d6ee40379fe8181c2dbd7a1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da895c90d6ee40379fe8181c2dbd7a1e already in progress: creating tablet
I20250625 07:52:29.837567 1161 tablet_replica.cc:331] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.837805 1161 raft_consensus.cc:2241] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.838032 1161 raft_consensus.cc:2270] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.838809 1161 ts_tablet_manager.cc:1905] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.841306 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a1baf94a051e485eaf4c0e074e147252 (table default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]) successfully deleted
W20250625 07:52:29.844364 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 37fe97b46b874771ae5ca8c849720118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37fe97b46b874771ae5ca8c849720118 already in progress: creating tablet
I20250625 07:52:29.844233 1163 tablet_bootstrap.cc:654] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.846171 1160 tablet_replica.cc:331] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:29.847396 1160 raft_consensus.cc:2241] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.848030 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_24fa140ea2d042a8ac731454569eebcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.848387 1160 raft_consensus.cc:2270] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:29.848600 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24fa140ea2d042a8ac731454569eebcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.850102 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 21658216da3544a2b8b8f39667782ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:13 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.850406 1160 ts_tablet_manager.cc:1905] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.854542 2045 raft_consensus.cc:491] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.854712 2045 raft_consensus.cc:513] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.855446 2045 leader_election.cc:290] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:29.856765 713 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:29.857237 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "743123ffcf8240dc82bea7178559b7ea" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.861969 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_602ca03b4f824a7686588dc776b8bed6" } new_table_name: "default.Table_b297f8cfc30a4f8cabed1ece38c77f00" modify_external_catalogs: true
I20250625 07:52:29.863471 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b2d591e4706149daa6e50d9466ea91ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.863958 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2d591e4706149daa6e50d9466ea91ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:29.865180 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2bcee2c83054f538c1f3ea05140685f consensus state has no leader
I20250625 07:52:29.865516 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c5934d5265ce47728b8a5007d7b1ae14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:29.865844 1162 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:29.865981 1162 ts_tablet_manager.cc:1397] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250625 07:52:29.866014 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5934d5265ce47728b8a5007d7b1ae14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.866912 1162 raft_consensus.cc:357] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.867084 1162 raft_consensus.cc:383] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.867153 1162 raft_consensus.cc:738] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.867420 1162 consensus_queue.cc:260] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.868094 1162 ts_tablet_manager.cc:1428] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 07:52:29.869058 713 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.869217 713 leader_election.cc:304] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.869454 2045 raft_consensus.cc:2747] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.871263 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fab2dff56bc7445abcd6b7c1cb525553
I20250625 07:52:29.872102 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_23c6bf383db34b55ac2b09f3327eced1" } new_table_name: "default.Table_ad8d548753e44f4ea9c2109f537c6194" modify_external_catalogs: true
I20250625 07:52:29.873322 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:29.829178 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:29.875906 876 tablet_service.cc:1515] Processing DeleteTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:29.877614 755 tablet_service.cc:1515] Processing DeleteTablet for tablet fab2dff56bc7445abcd6b7c1cb525553 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:29.877553 1160 ts_tablet_manager.cc:1918] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.881304 1160 log.cc:1199] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0929ed76622248d1a41f08f4258108cd
I20250625 07:52:29.882838 1166 ts_tablet_manager.cc:1918] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.882958 1166 log.cc:1199] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2771159858fc41a1ad930229986edd92
I20250625 07:52:29.882887 1160 ts_tablet_manager.cc:1939] T 0929ed76622248d1a41f08f4258108cd P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:29.883248 1166 ts_tablet_manager.cc:1939] T 2771159858fc41a1ad930229986edd92 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:29.885852 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_210190c86c3c4587986ad321a3d67e0b" } new_table_name: "default.Table_f6ad13e139c64d20abeb28e8584f0781" modify_external_catalogs: true
I20250625 07:52:29.886024 1162 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:29.887490 1162 tablet_bootstrap.cc:654] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.888486 1163 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:29.888723 1163 ts_tablet_manager.cc:1397] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.003s
I20250625 07:52:29.889547 1163 raft_consensus.cc:357] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.890458 1158 tablet_bootstrap.cc:492] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:29.891150 1158 ts_tablet_manager.cc:1397] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20250625 07:52:29.892315 1158 raft_consensus.cc:357] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.892537 1158 raft_consensus.cc:383] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.892652 1158 raft_consensus.cc:738] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.892953 1158 consensus_queue.cc:260] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.893625 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2771159858fc41a1ad930229986edd92 (table default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]) successfully deleted
I20250625 07:52:29.893905 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0929ed76622248d1a41f08f4258108cd (table default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]) successfully deleted
I20250625 07:52:29.894105 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca218f7ab1954346a19376d4d4898351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.894786 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca218f7ab1954346a19376d4d4898351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:29.895267 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a8a668fb466840e0a41749a4db9a1554 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a668fb466840e0a41749a4db9a1554 already in progress: creating tablet
W20250625 07:52:29.896565 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 113c05d81d2943e38bcd62bf83a46a2e consensus state has no leader
W20250625 07:52:29.896766 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c858e286fbba450e81c6cc55f0f2b613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c858e286fbba450e81c6cc55f0f2b613 already in progress: creating tablet
I20250625 07:52:29.897450 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b434115fc86e447b8270be77fca86aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.898216 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b434115fc86e447b8270be77fca86aa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:29.899397 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fab2dff56bc7445abcd6b7c1cb525553 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab2dff56bc7445abcd6b7c1cb525553 already in progress: creating tablet
I20250625 07:52:29.901314 1158 ts_tablet_manager.cc:1428] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 07:52:29.904665 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 883c5592cc014b8eb7d85f62242e5b02 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:29.904932 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ad78944352e4274acd8d9889b1d32b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:29.905109 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c8abe36eab5498eb9bc3a0355101dd6 has no consensus state
I20250625 07:52:29.905440 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bb7df6560d0044ebb0868389149f43a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:29.905968 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb7df6560d0044ebb0868389149f43a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:29.911090 2028 raft_consensus.cc:491] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.911262 2028 raft_consensus.cc:513] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.911839 2028 leader_election.cc:290] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:29.912882 986 leader_election.cc:343] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:29.914263 594 catalog_manager.cc:5582] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:29.914673 594 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:29.915030 594 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
W20250625 07:52:29.915966 985 leader_election.cc:343] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:29.916087 985 leader_election.cc:304] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:29.916313 2028 raft_consensus.cc:2747] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.916874 1016 tablet_service.cc:1468] Processing CreateTablet for tablet ba49ad0cb12d4f738288aea215dbda87 (DEFAULT_TABLE table=default.Table_3ddaab806779471489af8e38d15a75d1 [id=c63c34f09b0e4186af80ff305e3fc26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.917281 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba49ad0cb12d4f738288aea215dbda87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.919745 1158 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:29.921582 1158 tablet_bootstrap.cc:654] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.922699 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 (DEFAULT_TABLE table=default.Table_ad8d548753e44f4ea9c2109f537c6194 [id=d12fa195bc1b47b09c5238667a7d30db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.922816 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f5ed527f82564d8593e60df02137f529 (DEFAULT_TABLE table=default.Table_7b5cf53827f74c389a995bd409c38087 [id=0b8d7aa1965b42519fe9a76d4d795425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.923071 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8abe36eab5498eb9bc3a0355101dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.923144 1163 raft_consensus.cc:383] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:29.923218 1163 raft_consensus.cc:738] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:29.923422 1163 consensus_queue.cc:260] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.923571 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ed527f82564d8593e60df02137f529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.923910 1163 ts_tablet_manager.cc:1428] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250625 07:52:29.924197 1163 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:29.924993 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 34adc67622194ebeadd6b3b1163b2849 (DEFAULT_TABLE table=default.Table_066619da90c542d0a712b3e36629063b [id=1771a5f98e4d44a1a9bf76db13bea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.925361 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34adc67622194ebeadd6b3b1163b2849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.925738 1163 tablet_bootstrap.cc:654] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:29.926294 1161 ts_tablet_manager.cc:1918] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.926388 1161 log.cc:1199] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2a9945f38ef148caac59acaadcb2ef4c
I20250625 07:52:29.926668 1161 ts_tablet_manager.cc:1939] T 2a9945f38ef148caac59acaadcb2ef4c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:29.917028 876 tablet_service.cc:1468] Processing CreateTablet for tablet ba49ad0cb12d4f738288aea215dbda87 (DEFAULT_TABLE table=default.Table_3ddaab806779471489af8e38d15a75d1 [id=c63c34f09b0e4186af80ff305e3fc26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.929292 2030 raft_consensus.cc:491] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.929421 2030 raft_consensus.cc:513] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.912483 1166 tablet_replica.cc:331] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:29.929837 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0e621427f14e48a497fcd6d20346cee7" } new_table_name: "default.Table_bb809906074c4cddb2d0f447dd235592" modify_external_catalogs: true
I20250625 07:52:29.930100 2030 leader_election.cc:290] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.919382 879 tablet_service.cc:1468] Processing CreateTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 (DEFAULT_TABLE table=default.Table_ad8d548753e44f4ea9c2109f537c6194 [id=d12fa195bc1b47b09c5238667a7d30db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.931106 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:29.931322 1166 raft_consensus.cc:2241] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.931491 1166 raft_consensus.cc:2270] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.931684 887 tablet_service.cc:1468] Processing CreateTablet for tablet 34adc67622194ebeadd6b3b1163b2849 (DEFAULT_TABLE table=default.Table_066619da90c542d0a712b3e36629063b [id=1771a5f98e4d44a1a9bf76db13bea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.932034 1166 ts_tablet_manager.cc:1905] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:29.932088 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba49ad0cb12d4f738288aea215dbda87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.932832 755 tablet_service.cc:1468] Processing CreateTablet for tablet ba49ad0cb12d4f738288aea215dbda87 (DEFAULT_TABLE table=default.Table_3ddaab806779471489af8e38d15a75d1 [id=c63c34f09b0e4186af80ff305e3fc26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.932971 753 tablet_service.cc:1468] Processing CreateTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 (DEFAULT_TABLE table=default.Table_ad8d548753e44f4ea9c2109f537c6194 [id=d12fa195bc1b47b09c5238667a7d30db]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:29.933230 851 leader_election.cc:343] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.932971 757 tablet_service.cc:1468] Processing CreateTablet for tablet 34adc67622194ebeadd6b3b1163b2849 (DEFAULT_TABLE table=default.Table_066619da90c542d0a712b3e36629063b [id=1771a5f98e4d44a1a9bf76db13bea3ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.933445 749 tablet_service.cc:1468] Processing CreateTablet for tablet f5ed527f82564d8593e60df02137f529 (DEFAULT_TABLE table=default.Table_7b5cf53827f74c389a995bd409c38087 [id=0b8d7aa1965b42519fe9a76d4d795425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.933212 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba49ad0cb12d4f738288aea215dbda87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.933941 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.934158 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8abe36eab5498eb9bc3a0355101dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.934326 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ed527f82564d8593e60df02137f529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.921310 882 tablet_service.cc:1468] Processing CreateTablet for tablet f5ed527f82564d8593e60df02137f529 (DEFAULT_TABLE table=default.Table_7b5cf53827f74c389a995bd409c38087 [id=0b8d7aa1965b42519fe9a76d4d795425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.934525 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34adc67622194ebeadd6b3b1163b2849. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:29.934603 850 leader_election.cc:343] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.934700 850 leader_election.cc:304] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.935087 1793 raft_consensus.cc:2747] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.935850 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8abe36eab5498eb9bc3a0355101dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.940964 1793 raft_consensus.cc:491] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.941126 1793 raft_consensus.cc:513] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.941838 1793 leader_election.cc:290] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.942575 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "019959e9abed46fbbbb4755cb01cef44" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:29.943609 851 leader_election.cc:343] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.948109 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5ed527f82564d8593e60df02137f529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.948920 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34adc67622194ebeadd6b3b1163b2849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.957283 1161 tablet_replica.cc:331] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:29.957343 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "019959e9abed46fbbbb4755cb01cef44" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.957546 1161 raft_consensus.cc:2241] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:29.957705 1161 raft_consensus.cc:2270] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:29.958506 1161 ts_tablet_manager.cc:1905] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:29.958611 850 leader_election.cc:343] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.958717 850 leader_election.cc:304] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.959149 1793 raft_consensus.cc:2747] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.968806 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2a9945f38ef148caac59acaadcb2ef4c (table default.Table_4f645096238747a1a97bcd673b5438fd [id=836816eb909641daba87bdae20e0fd24]) successfully deleted
W20250625 07:52:29.969165 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1c1a1685d87e45e1857af0b3532e7b10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c1a1685d87e45e1857af0b3532e7b10 already in progress: creating tablet
I20250625 07:52:29.977010 1793 raft_consensus.cc:491] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.977150 1793 raft_consensus.cc:513] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:29.977671 1793 leader_election.cc:290] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.978214 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "871b603cb92249c6b0f16c825bf46232" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:29.978214 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "871b603cb92249c6b0f16c825bf46232" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:29.978688 851 leader_election.cc:343] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:29.978919 850 leader_election.cc:343] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:29.979034 850 leader_election.cc:304] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.979398 1793 raft_consensus.cc:2747] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.989507 2045 raft_consensus.cc:491] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:29.989691 2045 raft_consensus.cc:513] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:29.990315 2045 leader_election.cc:290] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:29.991029 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7b755d95b14ecea7a691003455f381" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:29.991536 713 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:29.991740 713 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: be7b755d95b14ecea7a691003455f381
I20250625 07:52:29.991837 713 leader_election.cc:304] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:29.992185 2045 raft_consensus.cc:2747] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:29.993227 1013 tablet_service.cc:1468] Processing CreateTablet for tablet a21fec7752884ca896213e4a5adcc8a7 (DEFAULT_TABLE table=default.Table_546234ae53e94032967f721a5e695354 [id=7b0cafe08e7b4481a23f21cc95c177c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.993432 755 tablet_service.cc:1468] Processing CreateTablet for tablet a21fec7752884ca896213e4a5adcc8a7 (DEFAULT_TABLE table=default.Table_546234ae53e94032967f721a5e695354 [id=7b0cafe08e7b4481a23f21cc95c177c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.993614 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21fec7752884ca896213e4a5adcc8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.993799 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21fec7752884ca896213e4a5adcc8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.996994 879 tablet_service.cc:1468] Processing CreateTablet for tablet a21fec7752884ca896213e4a5adcc8a7 (DEFAULT_TABLE table=default.Table_546234ae53e94032967f721a5e695354 [id=7b0cafe08e7b4481a23f21cc95c177c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:29.997395 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21fec7752884ca896213e4a5adcc8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:29.998059 1166 ts_tablet_manager.cc:1918] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:29.998214 1166 log.cc:1199] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:29.998551 1166 ts_tablet_manager.cc:1939] T 743123ffcf8240dc82bea7178559b7ea P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:29.999656 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 743123ffcf8240dc82bea7178559b7ea (table default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]) successfully deleted
I20250625 07:52:30.001442 1166 tablet_replica.cc:331] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.001698 1166 raft_consensus.cc:2241] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:30.004283 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a41ffd566593499299aba5513ccb2f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ffd566593499299aba5513ccb2f30 already in progress: creating tablet
W20250625 07:52:30.004561 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f1acadde98f54b65830c57df55b06ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1acadde98f54b65830c57df55b06ec3 already in progress: creating tablet
I20250625 07:52:30.004866 1166 raft_consensus.cc:2270] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.005474 1166 ts_tablet_manager.cc:1905] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.007527 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 0de0641aa91e4de682efe31fe074f8e5 (DEFAULT_TABLE table=default.Table_da2cc3d0fc2a4550847f4606db8c777a [id=3790f27b85e542aea9d007f130bc5d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.008803 757 tablet_service.cc:1468] Processing CreateTablet for tablet 0de0641aa91e4de682efe31fe074f8e5 (DEFAULT_TABLE table=default.Table_da2cc3d0fc2a4550847f4606db8c777a [id=3790f27b85e542aea9d007f130bc5d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.009310 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de0641aa91e4de682efe31fe074f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.009479 749 tablet_service.cc:1468] Processing CreateTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 (DEFAULT_TABLE table=default.Table_40a9b063009d432680f575cbc3dfad3f [id=c52549ccea78473cb5ea1dd877293a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.009784 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac3e3937ef074eb78550f9dd4965c7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.010016 876 tablet_service.cc:1468] Processing CreateTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 (DEFAULT_TABLE table=default.Table_40a9b063009d432680f575cbc3dfad3f [id=c52549ccea78473cb5ea1dd877293a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.010373 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac3e3937ef074eb78550f9dd4965c7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.014428 887 tablet_service.cc:1468] Processing CreateTablet for tablet 52ba11ba58014387bfaa994a8029b75c (DEFAULT_TABLE table=default.Table_226f128a33ac44438af91af68b3859b6 [id=c9d7790b993c4ae5a6f58974a35e0b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.013541 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 52ba11ba58014387bfaa994a8029b75c (DEFAULT_TABLE table=default.Table_226f128a33ac44438af91af68b3859b6 [id=c9d7790b993c4ae5a6f58974a35e0b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.014820 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ba11ba58014387bfaa994a8029b75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.014402 1016 tablet_service.cc:1468] Processing CreateTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 (DEFAULT_TABLE table=default.Table_40a9b063009d432680f575cbc3dfad3f [id=c52549ccea78473cb5ea1dd877293a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.008224 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de0641aa91e4de682efe31fe074f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.015489 753 tablet_service.cc:1468] Processing CreateTablet for tablet 52ba11ba58014387bfaa994a8029b75c (DEFAULT_TABLE table=default.Table_226f128a33ac44438af91af68b3859b6 [id=c9d7790b993c4ae5a6f58974a35e0b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.015710 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ba11ba58014387bfaa994a8029b75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.015810 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ba11ba58014387bfaa994a8029b75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.016400 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac3e3937ef074eb78550f9dd4965c7b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.035591 1161 ts_tablet_manager.cc:1918] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.035813 1161 log.cc:1199] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c65f257ae653488a8b49b0b523d2cf82
I20250625 07:52:30.036211 1161 ts_tablet_manager.cc:1939] T c65f257ae653488a8b49b0b523d2cf82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.036758 1162 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.040967 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 33f56c267457451e864e277e110c68fa
W20250625 07:52:30.042299 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cadd41655404316b10939e46d370c76 consensus state has no leader
I20250625 07:52:30.042454 1162 ts_tablet_manager.cc:1397] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.009s
I20250625 07:52:30.042644 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b9c59f89eb80476cb447870b7c2b4251" } modify_external_catalogs: true
I20250625 07:52:30.042814 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b9c59f89eb80476cb447870b7c2b4251" } modify_external_catalogs: true
I20250625 07:52:30.043247 1162 raft_consensus.cc:357] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.043409 1162 raft_consensus.cc:383] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.043457 1162 raft_consensus.cc:738] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.044499 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b649eef37b514286a98a4bc732c1870a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.045050 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b649eef37b514286a98a4bc732c1870a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.045104 749 tablet_service.cc:1468] Processing CreateTablet for tablet d18ad92511024e1daab6cca07e5e06b5 (DEFAULT_TABLE table=default.Table_4dc958f1088e4e719b8afc3b90c3b8dc [id=d17d38e742f142cf9a08a7f2affdc373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.045476 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18ad92511024e1daab6cca07e5e06b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.046676 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 883c5592cc014b8eb7d85f62242e5b02
I20250625 07:52:30.046643 882 tablet_service.cc:1468] Processing CreateTablet for tablet 29cd4ae297194079a7dfaa17152e3947 (DEFAULT_TABLE table=default.Table_ae8736f8b7954519bedb73c2427040e6 [id=a7b9c60f3cc5423c844cc09d0a916447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.047097 588 catalog_manager.cc:6652] ReplaceTablet: tablet 883c5592cc014b8eb7d85f62242e5b02 deleted and replaced by tablet 86d9f99c3ed34b2794236282040c2c49
I20250625 07:52:30.048276 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d9f99c3ed34b2794236282040c2c49) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.049314 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.049495 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.049641 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.050254 1160 tablet_replica.cc:331] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:30.050437 1160 raft_consensus.cc:2241] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.050578 1160 raft_consensus.cc:2270] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:30.051039 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 883c5592cc014b8eb7d85f62242e5b02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883c5592cc014b8eb7d85f62242e5b02 already in progress: creating tablet
I20250625 07:52:30.051174 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d9f99c3ed34b2794236282040c2c49) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.051211 753 tablet_service.cc:1468] Processing CreateTablet for tablet 29cd4ae297194079a7dfaa17152e3947 (DEFAULT_TABLE table=default.Table_ae8736f8b7954519bedb73c2427040e6 [id=a7b9c60f3cc5423c844cc09d0a916447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.051369 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cd4ae297194079a7dfaa17152e3947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.051542 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cd4ae297194079a7dfaa17152e3947. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:30.057013 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d019a5abdbe452d98f3d30ce45417c0 consensus state has no leader
I20250625 07:52:30.057123 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ac3e3937ef074eb78550f9dd4965c7b7
I20250625 07:52:30.040642 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 34a338005a7b44b9b0fba7a7096cfcb8
I20250625 07:52:30.058084 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.059218 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.060804 753 tablet_service.cc:1515] Processing DeleteTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.061084 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.061167 1160 ts_tablet_manager.cc:1905] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.061295 879 tablet_service.cc:1468] Processing CreateTablet for tablet 0de0641aa91e4de682efe31fe074f8e5 (DEFAULT_TABLE table=default.Table_da2cc3d0fc2a4550847f4606db8c777a [id=3790f27b85e542aea9d007f130bc5d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.061620 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de0641aa91e4de682efe31fe074f8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.064268 876 tablet_service.cc:1468] Processing CreateTablet for tablet d18ad92511024e1daab6cca07e5e06b5 (DEFAULT_TABLE table=default.Table_4dc958f1088e4e719b8afc3b90c3b8dc [id=d17d38e742f142cf9a08a7f2affdc373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.064601 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18ad92511024e1daab6cca07e5e06b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.037570 1017 tablet_service.cc:1468] Processing CreateTablet for tablet d18ad92511024e1daab6cca07e5e06b5 (DEFAULT_TABLE table=default.Table_4dc958f1088e4e719b8afc3b90c3b8dc [id=d17d38e742f142cf9a08a7f2affdc373]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.041678 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 33f56c267457451e864e277e110c68fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.045068 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 29cd4ae297194079a7dfaa17152e3947 (DEFAULT_TABLE table=default.Table_ae8736f8b7954519bedb73c2427040e6 [id=a7b9c60f3cc5423c844cc09d0a916447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.047418 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 883c5592cc014b8eb7d85f62242e5b02 with delete_type TABLET_DATA_DELETED (replaced by tablet 86d9f99c3ed34b2794236282040c2c49) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.065186 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29cd4ae297194079a7dfaa17152e3947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.065243 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b1e0cba0d5f340a88739f9994705b2e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.065856 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1e0cba0d5f340a88739f9994705b2e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.066519 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 1da4f7cff4144223a1b5b5430c6abc23 (DEFAULT_TABLE table=default.Table_f7ccc476abc14686bbffff9afc893419 [id=b2b2743e12b84482a010a27a200bca74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.067941 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 731a30dc035146d584d11cbecd12baf0
I20250625 07:52:30.057804 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ac3e3937ef074eb78550f9dd4965c7b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.058910 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.069342 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 33f56c267457451e864e277e110c68fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.069835 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.071012 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8240eda72dae4b56a2f9173041146e83" } new_table_name: "default.Table_77cbef8c52394ac0b7bca7c76d7b0ec0" modify_external_catalogs: true
I20250625 07:52:30.071131 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 731a30dc035146d584d11cbecd12baf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.070538 1162 consensus_queue.cc:260] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.071636 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 731a30dc035146d584d11cbecd12baf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.074179 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b8146a0332fc4ef897cbf4a745c83b5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.074750 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8146a0332fc4ef897cbf4a745c83b5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.075918 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b047e3c49b234a66991bed76ac01e6c6" } modify_external_catalogs: true
I20250625 07:52:30.076845 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 731a30dc035146d584d11cbecd12baf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.077330 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1da4f7cff4144223a1b5b5430c6abc23 (DEFAULT_TABLE table=default.Table_f7ccc476abc14686bbffff9afc893419 [id=b2b2743e12b84482a010a27a200bca74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.078060 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_30292e2cad9c4c039fce627374179f83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:30.078433 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da4f7cff4144223a1b5b5430c6abc23. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:30.078604 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30292e2cad9c4c039fce627374179f83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.079988 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 33f56c267457451e864e277e110c68fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.084069 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321] alter complete (version 5)
I20250625 07:52:30.087253 755 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.088793 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c65f257ae653488a8b49b0b523d2cf82 (table default.Table_40651d9551314c82aab23007dc0c10d4 [id=a8e64c4d65e2478684d3f409fd2bb93a]) successfully deleted
I20250625 07:52:30.089231 1158 tablet_bootstrap.cc:492] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
W20250625 07:52:30.089521 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a41ffd566593499299aba5513ccb2f30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a41ffd566593499299aba5513ccb2f30 already in progress: creating tablet
I20250625 07:52:30.089910 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e046fc4755bc40769ba73e88edfe6b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.090342 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f1acadde98f54b65830c57df55b06ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1acadde98f54b65830c57df55b06ec3 already in progress: creating tablet
W20250625 07:52:30.090667 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e046fc4755bc40769ba73e88edfe6b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:30.090946 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 37fe97b46b874771ae5ca8c849720118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37fe97b46b874771ae5ca8c849720118 already in progress: creating tablet
I20250625 07:52:30.091187 1161 tablet_replica.cc:331] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.091364 1161 raft_consensus.cc:2241] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.091497 1161 raft_consensus.cc:2270] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.091710 1158 ts_tablet_manager.cc:1397] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.172s user 0.003s sys 0.009s
I20250625 07:52:30.085211 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1da4f7cff4144223a1b5b5430c6abc23 (DEFAULT_TABLE table=default.Table_f7ccc476abc14686bbffff9afc893419 [id=b2b2743e12b84482a010a27a200bca74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.097414 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da4f7cff4144223a1b5b5430c6abc23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.097791 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b4fdd7ed1d8d4f2d814d8e22fea75b12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.098342 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4fdd7ed1d8d4f2d814d8e22fea75b12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.088815 1793 raft_consensus.cc:491] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.099032 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dc7e4a1ebfbe4239a253f502af55d599" } new_table_name: "default.Table_d61d2d07fd9447c2bb23eb09f21fd864" modify_external_catalogs: true
I20250625 07:52:30.099097 1793 raft_consensus.cc:513] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.099741 1166 ts_tablet_manager.cc:1918] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.099797 1793 leader_election.cc:290] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.099854 1166 log.cc:1199] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:30.100162 1166 ts_tablet_manager.cc:1939] T 06e4bf78218f4b82b4f16f59b1704256 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.100597 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.101137 850 leader_election.cc:343] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.104408 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1afe90251c964e369d05e82c7db19326 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.114302 1158 raft_consensus.cc:357] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.115036 1158 raft_consensus.cc:383] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.115260 1158 raft_consensus.cc:738] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.116106 1158 consensus_queue.cc:260] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.118129 1158 ts_tablet_manager.cc:1428] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20250625 07:52:30.091981 1161 ts_tablet_manager.cc:1905] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.120863 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:30.125016 851 leader_election.cc:343] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.125164 851 leader_election.cc:304] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.125795 1793 raft_consensus.cc:2747] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.126609 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 06e4bf78218f4b82b4f16f59b1704256 (table default.Table_4f95753979a34ea994409ba5978b7186 [id=4ff84ce52a874a5792b79f35f2016f1b]) successfully deleted
I20250625 07:52:30.127422 1046 raft_consensus.cc:3058] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:30.132442 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da4f7cff4144223a1b5b5430c6abc23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.134151 1166 tablet_replica.cc:331] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.134331 1166 raft_consensus.cc:2241] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.134459 1166 raft_consensus.cc:2270] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.134959 1166 ts_tablet_manager.cc:1905] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.136855 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18ad92511024e1daab6cca07e5e06b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:30.140734 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 37fe97b46b874771ae5ca8c849720118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37fe97b46b874771ae5ca8c849720118 already in progress: creating tablet
I20250625 07:52:30.146881 1166 ts_tablet_manager.cc:1918] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.147063 1166 log.cc:1199] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:30.147429 1166 ts_tablet_manager.cc:1939] T cbdbd9230210455ba615d51f1a98267b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.148545 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet cbdbd9230210455ba615d51f1a98267b (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
I20250625 07:52:30.153800 1163 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:30.155637 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a8a668fb466840e0a41749a4db9a1554 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a668fb466840e0a41749a4db9a1554 already in progress: creating tablet
W20250625 07:52:30.156203 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c858e286fbba450e81c6cc55f0f2b613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c858e286fbba450e81c6cc55f0f2b613 already in progress: creating tablet
I20250625 07:52:30.155355 1162 ts_tablet_manager.cc:1428] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.113s user 0.003s sys 0.000s
I20250625 07:52:30.157985 1166 tablet_replica.cc:331] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.158213 1166 raft_consensus.cc:2241] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.157275 1163 ts_tablet_manager.cc:1397] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.233s user 0.006s sys 0.007s
I20250625 07:52:30.158560 1166 raft_consensus.cc:2270] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.162870 1162 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.169816 1166 ts_tablet_manager.cc:1905] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.174688 1158 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.185081 1163 raft_consensus.cc:357] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.192234 1158 tablet_bootstrap.cc:654] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.193951 1163 raft_consensus.cc:383] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.194164 1163 raft_consensus.cc:738] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.194986 1163 consensus_queue.cc:260] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.202548 1162 tablet_bootstrap.cc:654] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.204159 1163 ts_tablet_manager.cc:1428] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.045s user 0.003s sys 0.001s
I20250625 07:52:30.212149 1163 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.213441 1850 raft_consensus.cc:491] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.213590 1850 raft_consensus.cc:513] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.214226 1850 leader_election.cc:290] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.214799 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6c3be49be14ecbb3c26ef107ede015" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.215167 713 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: fc6c3be49be14ecbb3c26ef107ede015
W20250625 07:52:30.215546 713 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.215636 713 leader_election.cc:304] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.215840 1850 raft_consensus.cc:2747] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.220898 1163 tablet_bootstrap.cc:654] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.215404 2062 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:30.228598 2028 raft_consensus.cc:491] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.228613 1793 raft_consensus.cc:491] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.228778 2028 raft_consensus.cc:513] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.228794 1793 raft_consensus.cc:513] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.229422 2028 leader_election.cc:290] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.229471 1793 leader_election.cc:290] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.230100 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:30.230211 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.231006 850 leader_election.cc:343] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:30.231367 985 leader_election.cc:343] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0024618293d74f559b04258f66616044
W20250625 07:52:30.231726 986 leader_election.cc:343] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0024618293d74f559b04258f66616044
I20250625 07:52:30.231844 986 leader_election.cc:304] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.232273 2028 raft_consensus.cc:2747] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:30.232570 851 leader_election.cc:343] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.232671 851 leader_election.cc:304] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.233153 1793 raft_consensus.cc:2747] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.233419 1793 raft_consensus.cc:491] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.233522 1793 raft_consensus.cc:513] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.234148 1793 leader_election.cc:290] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.234743 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2bb060cdd9a48688967989d7491cfc1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:30.234881 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2bb060cdd9a48688967989d7491cfc1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:30.235203 850 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:30.235590 851 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.235677 851 leader_election.cc:304] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.236061 1793 raft_consensus.cc:2747] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.243505 1160 ts_tablet_manager.cc:1918] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.244911 1160 log.cc:1199] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:30.246480 1161 ts_tablet_manager.cc:1918] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.246762 1161 log.cc:1199] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0e4519b1fbf34ca8b33655f4d402aafc
I20250625 07:52:30.247597 1161 ts_tablet_manager.cc:1939] T 0e4519b1fbf34ca8b33655f4d402aafc P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.251253 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0e4519b1fbf34ca8b33655f4d402aafc (table default.Table_16fdbb4827074529aaa4f9a805a01468 [id=ecb7ab30f3e64337b3e2b4cc59858112]) successfully deleted
I20250625 07:52:30.252682 1161 tablet_replica.cc:331] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.259136 1162 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.260020 1158 tablet_bootstrap.cc:492] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.261013 1158 ts_tablet_manager.cc:1397] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.003s
I20250625 07:52:30.267107 1162 ts_tablet_manager.cc:1397] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.106s user 0.008s sys 0.000s
I20250625 07:52:30.274048 1158 raft_consensus.cc:357] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.274611 1162 raft_consensus.cc:357] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.254005 1161 raft_consensus.cc:2241] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.275913 1162 raft_consensus.cc:383] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.275652 1158 raft_consensus.cc:383] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.276859 1162 raft_consensus.cc:738] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.276926 1158 raft_consensus.cc:738] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.277612 1161 raft_consensus.cc:2270] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.281474 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:30.281716 1160 ts_tablet_manager.cc:1939] T 4167282133f94bfb9870bd08fc6594b4 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:30.297494 1161 ts_tablet_manager.cc:1905] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.279348 1162 consensus_queue.cc:260] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.299633 2028 raft_consensus.cc:491] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.299791 2028 raft_consensus.cc:513] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.300453 2028 leader_election.cc:290] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:30.301627 986 leader_election.cc:343] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:30.302317 1162 ts_tablet_manager.cc:1428] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20250625 07:52:30.279214 1158 consensus_queue.cc:260] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:30.303206 985 leader_election.cc:343] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:30.303316 985 leader_election.cc:304] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.303757 2028 raft_consensus.cc:2747] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.309908 1158 ts_tablet_manager.cc:1428] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.042s user 0.001s sys 0.002s
I20250625 07:52:30.316548 1163 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.316812 1163 ts_tablet_manager.cc:1397] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.000s
I20250625 07:52:30.317723 1163 raft_consensus.cc:357] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.317983 1163 raft_consensus.cc:383] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.318086 1163 raft_consensus.cc:738] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.318538 1163 consensus_queue.cc:260] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.319110 1158 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.324981 1162 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.334759 1793 consensus_queue.cc:1035] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:30.353515 1158 tablet_bootstrap.cc:654] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.355050 2045 raft_consensus.cc:491] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.355230 2045 raft_consensus.cc:513] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.355978 2045 leader_election.cc:290] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.355782 1163 ts_tablet_manager.cc:1428] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.039s user 0.004s sys 0.000s
I20250625 07:52:30.357124 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1afe90251c964e369d05e82c7db19326" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.357971 713 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:30.358346 713 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1afe90251c964e369d05e82c7db19326
I20250625 07:52:30.358464 713 leader_election.cc:304] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.358873 1850 raft_consensus.cc:2747] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.362962 1850 raft_consensus.cc:491] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.363134 1850 raft_consensus.cc:513] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.363729 1850 leader_election.cc:290] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.364503 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd728ffc2a274deea3fad533d1eaa6da" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.365150 713 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:30.366091 713 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:30.366218 713 leader_election.cc:304] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.366580 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4167282133f94bfb9870bd08fc6594b4 (table default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]) successfully deleted
I20250625 07:52:30.366680 1850 raft_consensus.cc:2747] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.367439 1160 tablet_replica.cc:331] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:30.373179 1162 tablet_bootstrap.cc:654] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.425642 1160 raft_consensus.cc:2241] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:30.428726 850 consensus_peers.cc:487] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:30.428413 1160 raft_consensus.cc:2270] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.433557 1163 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.445022 1161 ts_tablet_manager.cc:1918] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.445374 1161 log.cc:1199] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2d8d3300befc4f419070ae84725ee96a
I20250625 07:52:30.446382 1161 ts_tablet_manager.cc:1939] T 2d8d3300befc4f419070ae84725ee96a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.447741 1162 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.449879 1793 raft_consensus.cc:491] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.450091 1793 raft_consensus.cc:513] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.451368 1793 leader_election.cc:290] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.451848 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:30.451782 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:30.452409 851 leader_election.cc:343] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:30.452615 850 leader_election.cc:343] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.452728 850 leader_election.cc:304] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.448099 1162 ts_tablet_manager.cc:1397] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.005s
I20250625 07:52:30.453274 1793 raft_consensus.cc:2747] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.455315 1162 raft_consensus.cc:357] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.455802 1162 raft_consensus.cc:383] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.456040 1162 raft_consensus.cc:738] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.456591 1162 consensus_queue.cc:260] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.467399 1158 tablet_bootstrap.cc:492] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.467373 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2d8d3300befc4f419070ae84725ee96a (table default.Table_dc62a435b6d44031bc3e5a60a3914e8f [id=f24c46ab73494d2b958f0fa750eadc63]) successfully deleted
I20250625 07:52:30.467531 1158 ts_tablet_manager.cc:1397] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.153s user 0.003s sys 0.003s
I20250625 07:52:30.468324 1158 raft_consensus.cc:357] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.468498 1158 raft_consensus.cc:383] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.468554 1158 raft_consensus.cc:738] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.468767 1158 consensus_queue.cc:260] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.469236 1158 ts_tablet_manager.cc:1428] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:30.469822 1166 ts_tablet_manager.cc:1918] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.469977 1166 log.cc:1199] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0f566cc9cdd447798a5cb408ef9cd35a
I20250625 07:52:30.470286 1166 ts_tablet_manager.cc:1939] T 0f566cc9cdd447798a5cb408ef9cd35a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.471117 1166 tablet_replica.cc:331] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.472002 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0f566cc9cdd447798a5cb408ef9cd35a (table default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]) successfully deleted
I20250625 07:52:30.479866 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a3670cadfa2841949705c65c9b6c0927 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.480026 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.480059 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6aceb5dc89a54074b4b33990008cb85e" } new_table_name: "default.Table_ab055810281b424ba5691e027be9ebe3" modify_external_catalogs: true
I20250625 07:52:30.484354 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
I20250625 07:52:30.485716 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:30.486433 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 4b4627f4da6b4542aff1e44157087b14 (table default.Table_d5657e03390941c28681972134ee0159 [id=618d684f882f425dab0d9a7395deb004]). Expected version 3 got 0
I20250625 07:52:30.486668 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]). Expected version 1 got 0
I20250625 07:52:30.488250 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.488516 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b047e3c49b234a66991bed76ac01e6c6" } modify_external_catalogs: true
I20250625 07:52:30.488641 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.489243 1162 ts_tablet_manager.cc:1428] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20250625 07:52:30.489419 1161 tablet_replica.cc:331] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.489609 1161 raft_consensus.cc:2241] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.489787 1161 raft_consensus.cc:2270] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.490443 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b333f5398ea94718a9f9c3208775b51d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.491309 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b333f5398ea94718a9f9c3208775b51d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.494851 879 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.496884 1166 raft_consensus.cc:2241] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.497152 1166 raft_consensus.cc:2270] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.502374 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f196473315441c6bf8b5173ed46f833 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.502588 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 651c74bf9a9b47d6ab5a875a5c304783 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.503756 1793 raft_consensus.cc:491] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.503906 1793 raft_consensus.cc:513] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.505028 1158 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.508378 1158 tablet_bootstrap.cc:654] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.509388 1166 ts_tablet_manager.cc:1905] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.512312 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:30.512864 1160 ts_tablet_manager.cc:1905] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:30.512967 851 leader_election.cc:343] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.513361 1793 leader_election.cc:290] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.513759 1162 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.513760 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:30.514547 1161 ts_tablet_manager.cc:1905] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:30.517976 850 leader_election.cc:343] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.518127 850 leader_election.cc:304] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.518697 2028 raft_consensus.cc:491] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.518823 2028 raft_consensus.cc:513] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.519337 2028 leader_election.cc:290] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:30.520602 986 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 651c74bf9a9b47d6ab5a875a5c304783
W20250625 07:52:30.521015 985 leader_election.cc:343] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 651c74bf9a9b47d6ab5a875a5c304783
I20250625 07:52:30.521131 985 leader_election.cc:304] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.521605 2028 raft_consensus.cc:2747] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.523959 1162 tablet_bootstrap.cc:654] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:30.492368 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73ec635d7d5242e8804dd3527457a540 consensus state has no leader
I20250625 07:52:30.524801 1793 raft_consensus.cc:2747] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.493419 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56a7bd0b475e451b8afa385ff9915ba5" } modify_external_catalogs: true
W20250625 07:52:30.493950 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5755b609b94f446f83341c16870ac1c1 consensus state has no leader
I20250625 07:52:30.525281 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56a7bd0b475e451b8afa385ff9915ba5" } modify_external_catalogs: true
I20250625 07:52:30.527120 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
W20250625 07:52:30.494081 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e68b52183b3f4b11b81d7cee20391953 consensus state has no leader
I20250625 07:52:30.496119 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b7c1b9f303dd44bda543ea0259d42097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:30.527671 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dbc79667956e45a6963ad166c7a8440d" } new_table_name: "default.Table_78b9fbe10b874ebd97522e5514939820" modify_external_catalogs: true
I20250625 07:52:30.496330 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3245e577c1b349f0b11133c6d1363526" } new_table_name: "default.Table_3fe2fe19b9c14eec9c5ab1fb6f50560a" modify_external_catalogs: true
I20250625 07:52:30.498894 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_22d71b8b5d5442c2b676651f64104641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.530421 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d71b8b5d5442c2b676651f64104641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.530817 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb7df6560d0044ebb0868389149f43a2" } modify_external_catalogs: true
I20250625 07:52:30.531162 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb7df6560d0044ebb0868389149f43a2" } modify_external_catalogs: true
I20250625 07:52:30.497779 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d0a75906385744e5ba6938f451a2d7ec
I20250625 07:52:30.532429 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d0a75906385744e5ba6938f451a2d7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.533262 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d0a75906385744e5ba6938f451a2d7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.533762 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5ddc30bfdeea430e8109fea6979694a5" } modify_external_catalogs: true
W20250625 07:52:30.527768 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7c1b9f303dd44bda543ea0259d42097 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.534379 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5ddc30bfdeea430e8109fea6979694a5" } modify_external_catalogs: true
I20250625 07:52:30.517980 1163 tablet_bootstrap.cc:654] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.537823 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9f4359fd842a4760bae75169e8914f1c
I20250625 07:52:30.503392 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bb75ba8ad3674db084f085ca2c573442"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.539162 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb75ba8ad3674db084f085ca2c573442 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.539371 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f4359fd842a4760bae75169e8914f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.540096 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f4359fd842a4760bae75169e8914f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.541616 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a950eac635448a1ac3562d75314e352" } modify_external_catalogs: true
I20250625 07:52:30.541839 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a950eac635448a1ac3562d75314e352" } modify_external_catalogs: true
W20250625 07:52:30.544296 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ca37dbcdd3a4a50a99667d221ef2bcb consensus state has no leader
I20250625 07:52:30.544562 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 3bed3f04c60249aaa389c26788974f58 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:30.545801 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fe1ce368cbe14d83a41437e55615e037
I20250625 07:52:30.546679 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5986bb52335c46bf845c48cb044495b1
I20250625 07:52:30.547421 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 5986bb52335c46bf845c48cb044495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.548010 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 5986bb52335c46bf845c48cb044495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.555672 1158 tablet_bootstrap.cc:492] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.555838 1158 ts_tablet_manager.cc:1397] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20250625 07:52:30.556619 1158 raft_consensus.cc:357] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.556824 1158 raft_consensus.cc:383] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.556895 1158 raft_consensus.cc:738] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.557108 1158 consensus_queue.cc:260] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.557392 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7ef2eb85358b414c9755f81f97beaddb
I20250625 07:52:30.558164 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef2eb85358b414c9755f81f97beaddb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.559041 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef2eb85358b414c9755f81f97beaddb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.560026 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:30.563005 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 802c22a6346442c88abbeb4b9ef20c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:30.566923 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 3ca37dbcdd3a4a50a99667d221ef2bcb (table default.Table_78b9fbe10b874ebd97522e5514939820 [id=1b9ac6804b0c4c35b721470779b09901], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3ca37dbcdd3a4a50a99667d221ef2bcb consensus state has no leader
I20250625 07:52:30.574414 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a
I20250625 07:52:30.575513 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a with delete_type TABLET_DATA_DELETED (replaced by tablet 82d661545e924905afa3d08eccac2e65) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.576148 1163 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.576279 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a with delete_type TABLET_DATA_DELETED (replaced by tablet 82d661545e924905afa3d08eccac2e65) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.576365 1158 ts_tablet_manager.cc:1428] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20250625 07:52:30.576756 1158 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.576308 1163 ts_tablet_manager.cc:1397] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.169s user 0.005s sys 0.001s
I20250625 07:52:30.577520 1163 raft_consensus.cc:357] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.578631 1163 raft_consensus.cc:383] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.578701 1163 raft_consensus.cc:738] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.578894 1163 consensus_queue.cc:260] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.579396 1163 ts_tablet_manager.cc:1428] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:30.580783 1793 raft_consensus.cc:491] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.580900 1793 raft_consensus.cc:513] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.581000 1160 ts_tablet_manager.cc:1918] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.581456 1793 leader_election.cc:290] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.581728 1163 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.582440 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccddd630a82341cba7a857e67a349fe0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.582877 850 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.583158 1163 tablet_bootstrap.cc:654] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.584175 597 catalog_manager.cc:6652] ReplaceTablet: tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a deleted and replaced by tablet 82d661545e924905afa3d08eccac2e65
W20250625 07:52:30.586292 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 494451c0c2514b08ae6be20391a7bcf8 consensus state has no leader
I20250625 07:52:30.590612 1158 tablet_bootstrap.cc:654] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.534385 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d0a75906385744e5ba6938f451a2d7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.538597 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f4359fd842a4760bae75169e8914f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.548483 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 5986bb52335c46bf845c48cb044495b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.559288 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef2eb85358b414c9755f81f97beaddb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.576818 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a with delete_type TABLET_DATA_DELETED (replaced by tablet 82d661545e924905afa3d08eccac2e65) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.582330 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccddd630a82341cba7a857e67a349fe0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:30.585510 1166 ts_tablet_manager.cc:1918] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.591732 1166 log.cc:1199] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:30.592082 1166 ts_tablet_manager.cc:1939] T fc6c3be49be14ecbb3c26ef107ede015 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:30.592092 851 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.592202 851 leader_election.cc:304] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.592747 1793 raft_consensus.cc:2747] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.593485 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet fc6c3be49be14ecbb3c26ef107ede015 (table default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]) successfully deleted
I20250625 07:52:30.594291 1166 tablet_replica.cc:331] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:30.594158 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fab2dff56bc7445abcd6b7c1cb525553 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab2dff56bc7445abcd6b7c1cb525553 already in progress: creating tablet
I20250625 07:52:30.594491 1166 raft_consensus.cc:2241] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.594651 1166 raft_consensus.cc:2270] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:30.594661 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 883c5592cc014b8eb7d85f62242e5b02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883c5592cc014b8eb7d85f62242e5b02 already in progress: creating tablet
W20250625 07:52:30.594980 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ac3e3937ef074eb78550f9dd4965c7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac3e3937ef074eb78550f9dd4965c7b7 already in progress: creating tablet
I20250625 07:52:30.595176 1166 ts_tablet_manager.cc:1905] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:30.595356 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 34a338005a7b44b9b0fba7a7096cfcb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34a338005a7b44b9b0fba7a7096cfcb8 already in progress: creating tablet
W20250625 07:52:30.595623 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 731a30dc035146d584d11cbecd12baf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 731a30dc035146d584d11cbecd12baf0 already in progress: creating tablet
W20250625 07:52:30.595868 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 33f56c267457451e864e277e110c68fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f56c267457451e864e277e110c68fa already in progress: creating tablet
I20250625 07:52:30.581102 1160 log.cc:1199] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:30.597105 1160 ts_tablet_manager.cc:1939] T c50c35cac8e94e088cdacb664b8adfe1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:30.597676 1160 tablet_replica.cc:331] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:30.597844 1160 raft_consensus.cc:2241] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.597816 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c50c35cac8e94e088cdacb664b8adfe1 (table default.Table_f6e3be4354c248d8abb373c9f0b5c6c9 [id=860ff357dc3a4a2dad9d850c113cf464]) successfully deleted
I20250625 07:52:30.597981 1160 raft_consensus.cc:2270] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.598578 1160 ts_tablet_manager.cc:1905] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.605687 1163 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.605842 1163 ts_tablet_manager.cc:1397] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20250625 07:52:30.606561 1163 raft_consensus.cc:357] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.606789 1163 raft_consensus.cc:383] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.606850 1163 raft_consensus.cc:738] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.607061 1163 consensus_queue.cc:260] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.607544 1163 ts_tablet_manager.cc:1428] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:30.612988 1163 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.614389 1163 tablet_bootstrap.cc:654] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.617228 1161 ts_tablet_manager.cc:1918] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.617367 1161 log.cc:1199] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0024618293d74f559b04258f66616044
I20250625 07:52:30.617660 1161 ts_tablet_manager.cc:1939] T 0024618293d74f559b04258f66616044 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.618371 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0024618293d74f559b04258f66616044 (table default.Table_73a04c95dfa64a86bd47f8aacefad904 [id=499784391ac245d28b2942aca33a79e4]) successfully deleted
I20250625 07:52:30.619470 1162 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.619640 1162 ts_tablet_manager.cc:1397] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.106s user 0.001s sys 0.003s
I20250625 07:52:30.621093 1161 tablet_replica.cc:331] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.621335 1162 raft_consensus.cc:357] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.621656 1162 raft_consensus.cc:383] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.621791 1162 raft_consensus.cc:738] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.621822 1161 raft_consensus.cc:2241] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.622030 1161 raft_consensus.cc:2270] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.622088 1162 consensus_queue.cc:260] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.622601 1162 ts_tablet_manager.cc:1428] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:30.623226 1158 tablet_bootstrap.cc:492] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.623356 1158 ts_tablet_manager.cc:1397] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250625 07:52:30.623719 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:30.624055 1158 raft_consensus.cc:357] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.624217 1158 raft_consensus.cc:383] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.624272 1158 raft_consensus.cc:738] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.624497 1158 consensus_queue.cc:260] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.624987 1162 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.626627 1162 tablet_bootstrap.cc:654] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.627751 1158 ts_tablet_manager.cc:1428] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:30.628019 1158 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.630564 1158 tablet_bootstrap.cc:654] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.631085 1161 ts_tablet_manager.cc:1905] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.633466 1163 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.633600 1163 ts_tablet_manager.cc:1397] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250625 07:52:30.612419 2077 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:30.634577 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b5f917cfd2384a72920847f8b89358b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.634403 1163 raft_consensus.cc:357] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.633612 2076 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:30.634837 1163 raft_consensus.cc:383] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.634893 1163 raft_consensus.cc:738] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.635080 1163 consensus_queue.cc:260] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.635555 1163 ts_tablet_manager.cc:1428] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:30.641544 1158 tablet_bootstrap.cc:492] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.641685 1158 ts_tablet_manager.cc:1397] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250625 07:52:30.642408 1158 raft_consensus.cc:357] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.642577 1158 raft_consensus.cc:383] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.642630 1158 raft_consensus.cc:738] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.642843 1158 consensus_queue.cc:260] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.643299 1158 ts_tablet_manager.cc:1428] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:30.644985 1163 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.645565 1850 raft_consensus.cc:491] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.645758 1850 raft_consensus.cc:513] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.646579 1163 tablet_bootstrap.cc:654] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.647150 1850 leader_election.cc:290] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.648190 1162 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.648336 1162 ts_tablet_manager.cc:1397] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20250625 07:52:30.648366 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1898ff5b7626478bb997f7a433477498" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.649101 713 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.649073 1162 raft_consensus.cc:357] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.650956 1162 raft_consensus.cc:383] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.651052 1162 raft_consensus.cc:738] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.651268 1162 consensus_queue.cc:260] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.651736 1162 ts_tablet_manager.cc:1428] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:30.654193 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ba24bbd0c4694930978a7750f07601d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.656780 1166 ts_tablet_manager.cc:1918] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.656936 1166 log.cc:1199] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/be7b755d95b14ecea7a691003455f381
I20250625 07:52:30.657295 1166 ts_tablet_manager.cc:1939] T be7b755d95b14ecea7a691003455f381 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:30.658704 713 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1898ff5b7626478bb997f7a433477498
I20250625 07:52:30.658829 713 leader_election.cc:304] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.659294 1850 raft_consensus.cc:2747] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.659549 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:30.662259 1162 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.664028 1162 tablet_bootstrap.cc:654] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.664862 1160 ts_tablet_manager.cc:1918] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.664990 1160 log.cc:1199] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:30.665283 1160 ts_tablet_manager.cc:1939] T ccddd630a82341cba7a857e67a349fe0 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:30.665758 1158 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.666710 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:30.667382 1158 tablet_bootstrap.cc:654] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.670509 1163 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.670641 1163 ts_tablet_manager.cc:1397] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20250625 07:52:30.671373 1163 raft_consensus.cc:357] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.671535 1163 raft_consensus.cc:383] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.671592 1163 raft_consensus.cc:738] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.671788 1163 consensus_queue.cc:260] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.672276 1163 ts_tablet_manager.cc:1428] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:30.673316 1166 tablet_replica.cc:331] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.673440 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet be7b755d95b14ecea7a691003455f381 (table default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]) successfully deleted
I20250625 07:52:30.673519 1166 raft_consensus.cc:2241] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.673691 1166 raft_consensus.cc:2270] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.679350 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0b1f254b07ab43f082eb0d6b1b02c48c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:30.679668 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8a81d60515d647b48496abdf8ef41532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.680084 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a81d60515d647b48496abdf8ef41532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:30.680083 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1f254b07ab43f082eb0d6b1b02c48c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.684345 1160 tablet_replica.cc:331] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:30.684540 1160 raft_consensus.cc:2241] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.684710 1160 raft_consensus.cc:2270] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.685230 1160 ts_tablet_manager.cc:1905] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.685400 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet ccddd630a82341cba7a857e67a349fe0 (table default.Table_80cbf681213f4716b1b7fd920d71b59c [id=3a15e8259a5f4b1283f0c106d55baff1]) successfully deleted
W20250625 07:52:30.685663 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ac3e3937ef074eb78550f9dd4965c7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac3e3937ef074eb78550f9dd4965c7b7 already in progress: creating tablet
W20250625 07:52:30.685971 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 34a338005a7b44b9b0fba7a7096cfcb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34a338005a7b44b9b0fba7a7096cfcb8 already in progress: creating tablet
W20250625 07:52:30.686172 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 33f56c267457451e864e277e110c68fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f56c267457451e864e277e110c68fa already in progress: creating tablet
I20250625 07:52:30.688479 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_37ecb82090564313862912776a682691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.689044 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37ecb82090564313862912776a682691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.693254 1161 ts_tablet_manager.cc:1918] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.693372 1161 log.cc:1199] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e7100ef72ff14c24be92c21c6eb8a9de
I20250625 07:52:30.693440 1163 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.693984 1161 ts_tablet_manager.cc:1939] T e7100ef72ff14c24be92c21c6eb8a9de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.695441 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e7100ef72ff14c24be92c21c6eb8a9de (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
I20250625 07:52:30.695643 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d45cd26e5aab475199c9c4b5ffb4d900"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.696180 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d45cd26e5aab475199c9c4b5ffb4d900 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:30.697171 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a8a668fb466840e0a41749a4db9a1554 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a668fb466840e0a41749a4db9a1554 already in progress: creating tablet
W20250625 07:52:30.698274 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c858e286fbba450e81c6cc55f0f2b613 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c858e286fbba450e81c6cc55f0f2b613 already in progress: creating tablet
I20250625 07:52:30.698961 1161 tablet_replica.cc:331] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.699798 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2f21fd181ae64529a96b8dd7b83c8cad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.700296 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f21fd181ae64529a96b8dd7b83c8cad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.699800 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_342146f029294bdaa3d7e68bc9e8d6b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.702267 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_342146f029294bdaa3d7e68bc9e8d6b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.702612 1162 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.702764 1162 ts_tablet_manager.cc:1397] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.003s
I20250625 07:52:30.703552 1162 raft_consensus.cc:357] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.703727 1162 raft_consensus.cc:383] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.703792 1162 raft_consensus.cc:738] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.704036 1162 consensus_queue.cc:260] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.705041 2028 raft_consensus.cc:491] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.705174 2028 raft_consensus.cc:513] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.704510 1162 ts_tablet_manager.cc:1428] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:30.705751 2028 leader_election.cc:290] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.705830 1162 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.707489 1162 tablet_bootstrap.cc:654] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:30.708403 986 leader_election.cc:343] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b5f917cfd2384a72920847f8b89358b9
W20250625 07:52:30.709285 985 leader_election.cc:343] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b5f917cfd2384a72920847f8b89358b9
I20250625 07:52:30.709395 985 leader_election.cc:304] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.709782 2028 raft_consensus.cc:2747] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.710069 1163 tablet_bootstrap.cc:654] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.711189 1166 ts_tablet_manager.cc:1905] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.710068 1161 raft_consensus.cc:2241] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.711417 1161 raft_consensus.cc:2270] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.712123 1161 ts_tablet_manager.cc:1905] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.715554 1158 tablet_bootstrap.cc:492] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.716871 1158 ts_tablet_manager.cc:1397] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20250625 07:52:30.723161 1160 ts_tablet_manager.cc:1918] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.723294 1160 log.cc:1199] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/871b603cb92249c6b0f16c825bf46232
I20250625 07:52:30.723589 1160 ts_tablet_manager.cc:1939] T 871b603cb92249c6b0f16c825bf46232 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:30.725718 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2aa50fc1fa7542bb8d872216447d73f9" } new_table_name: "default.Table_8f40c634b01a4ad19cb4a9c14cb057b7" modify_external_catalogs: true
I20250625 07:52:30.726559 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9a7acbd7be614574a81e31a0f68bdbed" } new_table_name: "default.Table_99eaf4f7bbec455ab0c9291fedfa221d" modify_external_catalogs: true
W20250625 07:52:30.732821 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64ca83369ffe46ea9bd756f20da936cf consensus state has no leader
I20250625 07:52:30.732831 2028 raft_consensus.cc:491] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.732964 2028 raft_consensus.cc:513] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.733585 2028 leader_election.cc:290] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.733788 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2231a0c930064911a051c1cf00c3d2e5" } new_table_name: "default.Table_b0da6c2d36ec4be598dcce766031462c" modify_external_catalogs: true
I20250625 07:52:30.734144 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b527d528908b40de9346966341c2f3fb" } modify_external_catalogs: true
I20250625 07:52:30.734316 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b527d528908b40de9346966341c2f3fb" } modify_external_catalogs: true
W20250625 07:52:30.734997 1989 log.cc:927] Time spent T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Append to log took a long time: real 0.055s user 0.003s sys 0.000s
W20250625 07:52:30.734952 986 leader_election.cc:343] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:30.733649 1158 raft_consensus.cc:357] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.737129 1158 raft_consensus.cc:383] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.738152 1158 raft_consensus.cc:738] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:30.739866 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 215021f96ce54429b8aa5e19668fba0b has no consensus state
I20250625 07:52:30.740103 1158 consensus_queue.cc:260] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.742085 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_00fc5f49db204824b892b2ae69796dca" } modify_external_catalogs: true
I20250625 07:52:30.742277 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_00fc5f49db204824b892b2ae69796dca" } modify_external_catalogs: true
I20250625 07:52:30.744621 1793 raft_consensus.cc:491] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.744799 1793 raft_consensus.cc:513] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.745373 1793 leader_election.cc:290] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.746153 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfacdde6a8ff4b6eac40d83c1d5b21f6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:30.746686 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_825f0fc01f0a47eaa92ad842fe4e7c68" } new_table_name: "default.Table_57825aadc5bf48329eb4259fe95fba1e" modify_external_catalogs: true
W20250625 07:52:30.746831 985 leader_election.cc:343] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:30.746941 985 leader_election.cc:304] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.747308 2028 raft_consensus.cc:2747] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.747656 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:30.748080 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c434f9d332484fc09394c912e328bf47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.748628 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c434f9d332484fc09394c912e328bf47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:30.748878 851 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.749589 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfacdde6a8ff4b6eac40d83c1d5b21f6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:30.750133 850 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:30.750236 850 leader_election.cc:304] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:30.750236 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_973a2ebd7372436296e18cdf46fc490f" } modify_external_catalogs: true
I20250625 07:52:30.750387 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_973a2ebd7372436296e18cdf46fc490f" } modify_external_catalogs: true
I20250625 07:52:30.752069 1793 raft_consensus.cc:2747] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.752584 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2e4dcd29c1264e7ead844712846cd17a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:30.752434 1158 ts_tablet_manager.cc:1428] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
W20250625 07:52:30.753121 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e4dcd29c1264e7ead844712846cd17a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.756964 596 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]). Expected version 1 got 0
I20250625 07:52:30.757772 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cf4e82b97e4649fc8e50c62a18f22d20" } modify_external_catalogs: true
I20250625 07:52:30.757508 1158 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.760236 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 871b603cb92249c6b0f16c825bf46232 (table default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]) successfully deleted
I20250625 07:52:30.761325 1163 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.761456 1163 ts_tablet_manager.cc:1397] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.002s
I20250625 07:52:30.762120 1163 raft_consensus.cc:357] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.762290 1163 raft_consensus.cc:383] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.762355 1163 raft_consensus.cc:738] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.762558 1163 consensus_queue.cc:260] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.762992 1163 ts_tablet_manager.cc:1428] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:30.763792 1162 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.763937 1162 ts_tablet_manager.cc:1397] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250625 07:52:30.764638 1162 raft_consensus.cc:357] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.764830 1162 raft_consensus.cc:383] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.764905 1162 raft_consensus.cc:738] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.765136 1162 consensus_queue.cc:260] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.765625 1162 ts_tablet_manager.cc:1428] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:30.765820 1166 ts_tablet_manager.cc:1918] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.765954 1166 log.cc:1199] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1afe90251c964e369d05e82c7db19326
I20250625 07:52:30.766287 2028 raft_consensus.cc:491] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.766283 1166 ts_tablet_manager.cc:1939] T 1afe90251c964e369d05e82c7db19326 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.766419 2028 raft_consensus.cc:513] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.767005 2028 leader_election.cc:290] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.768362 1161 ts_tablet_manager.cc:1918] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:30.768409 986 leader_election.cc:343] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7f196473315441c6bf8b5173ed46f833
I20250625 07:52:30.768476 1161 log.cc:1199] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/21658216da3544a2b8b8f39667782ffb
W20250625 07:52:30.768633 985 leader_election.cc:343] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7f196473315441c6bf8b5173ed46f833
I20250625 07:52:30.768738 985 leader_election.cc:304] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.769619 1161 ts_tablet_manager.cc:1939] T 21658216da3544a2b8b8f39667782ffb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.770198 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 21658216da3544a2b8b8f39667782ffb (table default.Table_5bc1b69194f54555909af8b886caf03d [id=ff3a0721feca4943b01e5350e2d2126b]) successfully deleted
W20250625 07:52:30.770821 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fab2dff56bc7445abcd6b7c1cb525553 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab2dff56bc7445abcd6b7c1cb525553 already in progress: creating tablet
I20250625 07:52:30.773077 2028 raft_consensus.cc:2747] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.774370 1161 tablet_replica.cc:331] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:30.774546 1161 raft_consensus.cc:2241] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.774698 1161 raft_consensus.cc:2270] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.775314 1162 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.776494 1161 ts_tablet_manager.cc:1905] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.777122 1162 tablet_bootstrap.cc:654] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.757947 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cf4e82b97e4649fc8e50c62a18f22d20" } modify_external_catalogs: true
I20250625 07:52:30.778596 1166 tablet_replica.cc:331] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.778790 1166 raft_consensus.cc:2241] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.778944 1166 raft_consensus.cc:2270] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.779434 1166 ts_tablet_manager.cc:1905] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.784389 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1afe90251c964e369d05e82c7db19326 (table default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]) successfully deleted
W20250625 07:52:30.786427 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d2f251602454a1eadf4e2803e226531 consensus state has no leader
I20250625 07:52:30.786870 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1860ecd4dd0340c39868730e3b641d4d" } modify_external_catalogs: true
I20250625 07:52:30.787032 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1860ecd4dd0340c39868730e3b641d4d" } modify_external_catalogs: true
W20250625 07:52:30.791760 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 731a30dc035146d584d11cbecd12baf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 731a30dc035146d584d11cbecd12baf0 already in progress: creating tablet
I20250625 07:52:30.793147 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4a44da8c8ff346118e917baceb47de8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:30.793414 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d77337df3beb48698fd178bc5b2783d2" } new_table_name: "default.Table_275c0087fed745359a026e8c16f028cf" modify_external_catalogs: true
W20250625 07:52:30.793838 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a44da8c8ff346118e917baceb47de8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.797246 1160 tablet_replica.cc:331] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:30.797488 1160 raft_consensus.cc:2241] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.797675 1160 raft_consensus.cc:2270] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.798285 1160 ts_tablet_manager.cc:1905] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.798362 1163 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:30.800031 1163 tablet_bootstrap.cc:654] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.800769 1158 tablet_bootstrap.cc:654] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.801543 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:30.802842 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_86b56ad960e1457ca74b3db973f6cbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.803390 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86b56ad960e1457ca74b3db973f6cbd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:30.806346 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80691f033eb44c6ba256b47acd868b70 consensus state has no leader
I20250625 07:52:30.807235 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.813560 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 6881336f1ff44e3cb7d1bc8937e503a4 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:30.819396 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:30.819895 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:30.820274 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:30.822809 1158 tablet_bootstrap.cc:492] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.822952 1158 ts_tablet_manager.cc:1397] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20250625 07:52:30.823652 1158 raft_consensus.cc:357] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.823817 1158 raft_consensus.cc:383] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.823859 1158 raft_consensus.cc:738] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.824044 1158 consensus_queue.cc:260] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.824491 1158 ts_tablet_manager.cc:1428] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:30.824846 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ad44a68d20774c8daa75b9252f90e442 (DEFAULT_TABLE table=default.Table_b434115fc86e447b8270be77fca86aa0 [id=c5ed72e80e5d4ae1a442c638c47e37ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.825242 1158 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.827343 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad44a68d20774c8daa75b9252f90e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.830648 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a7cb2bcdd07c42e680eb4f155212423f with delete_type TABLET_DATA_DELETED (replaced by tablet cbdbd9230210455ba615d51f1a98267b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.835228 1161 ts_tablet_manager.cc:1918] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.835388 1161 log.cc:1199] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7ad78944352e4274acd8d9889b1d32b8
I20250625 07:52:30.835764 1161 ts_tablet_manager.cc:1939] T 7ad78944352e4274acd8d9889b1d32b8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:30.836614 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87] alter complete (version 1)
I20250625 07:52:30.842195 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.842705 594 master_service.cc:948] ReplaceTablet: received request to replace tablet c0cc3ba09d5e4c3d88e74fffc97eea8b from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:30.842974 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.844767 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 683954dc7ecc4185847f46a05fd2c53d
I20250625 07:52:30.845397 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 652e07d46af74945bf504f7d9015a446
I20250625 07:52:30.847918 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 78a1bfd87f084d50bd2b602b89c18a2f
I20250625 07:52:30.848627 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b782414a682246feb2811b4e3bf8b5a2
I20250625 07:52:30.849262 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.849409 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.849874 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bedc38018eb6412b9e6ca3f0ee34cf87
I20250625 07:52:30.849922 1163 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:30.850054 1163 ts_tablet_manager.cc:1397] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
I20250625 07:52:30.854127 1160 ts_tablet_manager.cc:1918] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.854249 1160 log.cc:1199] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/019959e9abed46fbbbb4755cb01cef44
I20250625 07:52:30.854543 1160 ts_tablet_manager.cc:1939] T 019959e9abed46fbbbb4755cb01cef44 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:30.856967 876 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.857172 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6881336f1ff44e3cb7d1bc8937e503a4
I20250625 07:52:30.857372 876 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.857609 590 catalog_manager.cc:6652] ReplaceTablet: tablet 6881336f1ff44e3cb7d1bc8937e503a4 deleted and replaced by tablet 1748e9e9bdbc453da0c3ff120ae70ffd
I20250625 07:52:30.857664 876 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.858058 1166 ts_tablet_manager.cc:1918] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.858165 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 683954dc7ecc4185847f46a05fd2c53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.858193 1166 log.cc:1199] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1898ff5b7626478bb997f7a433477498
I20250625 07:52:30.858500 1166 ts_tablet_manager.cc:1939] T 1898ff5b7626478bb997f7a433477498 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.858572 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 652e07d46af74945bf504f7d9015a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.858987 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.859476 1166 tablet_replica.cc:331] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:30.859529 879 tablet_service.cc:1468] Processing CreateTablet for tablet ad44a68d20774c8daa75b9252f90e442 (DEFAULT_TABLE table=default.Table_b434115fc86e447b8270be77fca86aa0 [id=c5ed72e80e5d4ae1a442c638c47e37ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.859673 1166 raft_consensus.cc:2241] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:30.859854 1166 raft_consensus.cc:2270] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:30.860066 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 683954dc7ecc4185847f46a05fd2c53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.860319 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 652e07d46af74945bf504f7d9015a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.860445 1166 ts_tablet_manager.cc:1905] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:30.860564 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.860872 876 tablet_service.cc:1515] Processing DeleteTablet for tablet b782414a682246feb2811b4e3bf8b5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.861126 754 tablet_service.cc:1515] Processing DeleteTablet for tablet b782414a682246feb2811b4e3bf8b5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.861559 1162 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.861521 754 tablet_service.cc:1468] Processing CreateTablet for tablet a13b166a67304921b4fdc6ddc68f2035 (DEFAULT_TABLE table=default.Table_ca218f7ab1954346a19376d4d4898351 [id=f53cf9f0f9cc467599292043eb748831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.861725 1162 ts_tablet_manager.cc:1397] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20250625 07:52:30.862110 757 tablet_service.cc:1468] Processing CreateTablet for tablet 841d3a3c98af45339836d9849ff712fd (DEFAULT_TABLE table=default.Table_c5934d5265ce47728b8a5007d7b1ae14 [id=c87e661f1cc64bd7b1057299ff546e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.862478 755 tablet_service.cc:1515] Processing DeleteTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.862550 750 tablet_service.cc:1468] Processing CreateTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 (DEFAULT_TABLE table=default.Table_aef7709d3ff64d91a06278b1080f77ba [id=0d0af75b99604d7ca250d7f676d33032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.862824 749 tablet_service.cc:1468] Processing CreateTablet for tablet 31e06a96ef0847eb9568cb363e1ed6ab (DEFAULT_TABLE table=default.Table_dc4bf2e1a50b434f838b277b1edc3471 [id=9d48c316ea454f7b9c378166f151e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.863123 741 tablet_service.cc:1468] Processing CreateTablet for tablet 74c0e148f958453c8c26f8ba6460d13c (DEFAULT_TABLE table=default.Table_9c2f9426a94b42f9bbaf640f8b5917cb [id=9d09c12a144f4c50b32596ac1c38a757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.863310 755 tablet_service.cc:1468] Processing CreateTablet for tablet a3a76704c36b45e2afe54376e30f46fe (DEFAULT_TABLE table=default.Table_8c064e83460b4a58b1092ef7ae226598 [id=11d73cf528b94cfcbe8964a6d26e7d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.863227 1162 raft_consensus.cc:357] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.863538 1162 raft_consensus.cc:383] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.863842 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1748e9e9bdbc453da0c3ff120ae70ffd) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.863673 1162 raft_consensus.cc:738] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.864323 753 tablet_service.cc:1468] Processing CreateTablet for tablet ad44a68d20774c8daa75b9252f90e442 (DEFAULT_TABLE table=default.Table_b434115fc86e447b8270be77fca86aa0 [id=c5ed72e80e5d4ae1a442c638c47e37ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.864338 1162 consensus_queue.cc:260] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.865209 1162 ts_tablet_manager.cc:1428] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:30.865849 1162 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:30.867777 876 tablet_service.cc:1468] Processing CreateTablet for tablet a13b166a67304921b4fdc6ddc68f2035 (DEFAULT_TABLE table=default.Table_ca218f7ab1954346a19376d4d4898351 [id=f53cf9f0f9cc467599292043eb748831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.868584 882 tablet_service.cc:1515] Processing DeleteTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.869196 882 tablet_service.cc:1468] Processing CreateTablet for tablet 841d3a3c98af45339836d9849ff712fd (DEFAULT_TABLE table=default.Table_c5934d5265ce47728b8a5007d7b1ae14 [id=c87e661f1cc64bd7b1057299ff546e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.872795 887 tablet_service.cc:1468] Processing CreateTablet for tablet 31e06a96ef0847eb9568cb363e1ed6ab (DEFAULT_TABLE table=default.Table_dc4bf2e1a50b434f838b277b1edc3471 [id=9d48c316ea454f7b9c378166f151e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.873715 881 tablet_service.cc:1468] Processing CreateTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 (DEFAULT_TABLE table=default.Table_aef7709d3ff64d91a06278b1080f77ba [id=0d0af75b99604d7ca250d7f676d33032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.874109 885 tablet_service.cc:1468] Processing CreateTablet for tablet a3a76704c36b45e2afe54376e30f46fe (DEFAULT_TABLE table=default.Table_8c064e83460b4a58b1092ef7ae226598 [id=11d73cf528b94cfcbe8964a6d26e7d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.874451 877 tablet_service.cc:1468] Processing CreateTablet for tablet 74c0e148f958453c8c26f8ba6460d13c (DEFAULT_TABLE table=default.Table_9c2f9426a94b42f9bbaf640f8b5917cb [id=9d09c12a144f4c50b32596ac1c38a757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.876041 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1898ff5b7626478bb997f7a433477498 (table default.Table_8d5544c1432c4978ada968f7fcbfab59 [id=b0330ed5b54243c983112b2afaf66f1f]) successfully deleted
I20250625 07:52:30.877048 739 tablet_service.cc:1468] Processing CreateTablet for tablet 215021f96ce54429b8aa5e19668fba0b (DEFAULT_TABLE table=default.Table_b0da6c2d36ec4be598dcce766031462c [id=5c1c9c95e9454bfe8a9624b0ca62602e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.877323 746 tablet_service.cc:1468] Processing CreateTablet for tablet 67d5cb5b93764127b3cc056c4ecba2df (DEFAULT_TABLE table=default.Table_8c9a216aabd74a76990345c02842ea74 [id=f8fb9cf5b8824ec78a5bd07bd13f1428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.878250 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:30.838274 1158 tablet_bootstrap.cc:654] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.878459 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1748e9e9bdbc453da0c3ff120ae70ffd) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:30.879045 893 tablet_service.cc:1468] Processing CreateTablet for tablet 215021f96ce54429b8aa5e19668fba0b (DEFAULT_TABLE table=default.Table_b0da6c2d36ec4be598dcce766031462c [id=5c1c9c95e9454bfe8a9624b0ca62602e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.846282 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 652e07d46af74945bf504f7d9015a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.846915 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 683954dc7ecc4185847f46a05fd2c53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.848873 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a1bfd87f084d50bd2b602b89c18a2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.851555 1016 tablet_service.cc:1468] Processing CreateTablet for tablet a13b166a67304921b4fdc6ddc68f2035 (DEFAULT_TABLE table=default.Table_ca218f7ab1954346a19376d4d4898351 [id=f53cf9f0f9cc467599292043eb748831]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.851740 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet b782414a682246feb2811b4e3bf8b5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.879503 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b166a67304921b4fdc6ddc68f2035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.852020 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 841d3a3c98af45339836d9849ff712fd (DEFAULT_TABLE table=default.Table_c5934d5265ce47728b8a5007d7b1ae14 [id=c87e661f1cc64bd7b1057299ff546e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.852192 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.879735 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215021f96ce54429b8aa5e19668fba0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.852442 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 31e06a96ef0847eb9568cb363e1ed6ab (DEFAULT_TABLE table=default.Table_dc4bf2e1a50b434f838b277b1edc3471 [id=9d48c316ea454f7b9c378166f151e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.852766 1026 tablet_service.cc:1468] Processing CreateTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 (DEFAULT_TABLE table=default.Table_aef7709d3ff64d91a06278b1080f77ba [id=0d0af75b99604d7ca250d7f676d33032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.854811 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a3a76704c36b45e2afe54376e30f46fe (DEFAULT_TABLE table=default.Table_8c064e83460b4a58b1092ef7ae226598 [id=11d73cf528b94cfcbe8964a6d26e7d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.855077 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 74c0e148f958453c8c26f8ba6460d13c (DEFAULT_TABLE table=default.Table_9c2f9426a94b42f9bbaf640f8b5917cb [id=9d09c12a144f4c50b32596ac1c38a757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.855345 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 215021f96ce54429b8aa5e19668fba0b (DEFAULT_TABLE table=default.Table_b0da6c2d36ec4be598dcce766031462c [id=5c1c9c95e9454bfe8a9624b0ca62602e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.859179 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6881336f1ff44e3cb7d1bc8937e503a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 1748e9e9bdbc453da0c3ff120ae70ffd) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:30.870590 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 67d5cb5b93764127b3cc056c4ecba2df (DEFAULT_TABLE table=default.Table_8c9a216aabd74a76990345c02842ea74 [id=f8fb9cf5b8824ec78a5bd07bd13f1428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.881177 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d5cb5b93764127b3cc056c4ecba2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.875674 1018 tablet_service.cc:1468] Processing CreateTablet for tablet f0fe3a2f896c4100a4d484196094f40b (DEFAULT_TABLE table=default.Table_65094aac02f94466a02285f60f743c29 [id=2b8de9c3cc6a4e979eff7f97301b96f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.884841 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 841d3a3c98af45339836d9849ff712fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.885108 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c0e148f958453c8c26f8ba6460d13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.885373 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9eb3bcbece34f14a05dc7d95ca7ece2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.886993 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e06a96ef0847eb9568cb363e1ed6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.891626 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a76704c36b45e2afe54376e30f46fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.891932 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b6579bae985346f9bbb23891c04cbe8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.892601 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6579bae985346f9bbb23891c04cbe8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.893321 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad44a68d20774c8daa75b9252f90e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.894709 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d256e8aab118412d92953b648fe06283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.895251 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d256e8aab118412d92953b648fe06283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.898695 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_77374629d3b048f8a55d64335667dcea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.899230 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77374629d3b048f8a55d64335667dcea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.904755 1162 tablet_bootstrap.cc:654] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:30.908529 713 consensus_peers.cc:487] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:30.910445 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b166a67304921b4fdc6ddc68f2035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.910672 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fe3a2f896c4100a4d484196094f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.910858 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c0e148f958453c8c26f8ba6460d13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.913079 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2fa277660d76420a834714f10f39c3af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.913690 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fa277660d76420a834714f10f39c3af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.914835 2028 raft_consensus.cc:491] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.914979 2028 raft_consensus.cc:513] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.915525 2028 leader_election.cc:290] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:30.916098 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:30.916147 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fd497aa0b8c4838906524f4423c3756" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:30.916965 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a76704c36b45e2afe54376e30f46fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.917323 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_04ba3f20d1184e0c9f9deb21ea849289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.917918 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04ba3f20d1184e0c9f9deb21ea849289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.919109 1158 tablet_bootstrap.cc:492] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:30.919229 1158 ts_tablet_manager.cc:1397] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.008s
I20250625 07:52:30.919238 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0b3fc22526374bb1a70819a649be9dfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:30.919729 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b3fc22526374bb1a70819a649be9dfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:30.920032 1158 raft_consensus.cc:357] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.920176 1158 raft_consensus.cc:383] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.920238 1158 raft_consensus.cc:738] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.920413 1158 consensus_queue.cc:260] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.921089 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e06a96ef0847eb9568cb363e1ed6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.921066 1158 ts_tablet_manager.cc:1428] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:30.921460 1158 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:30.922865 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d5cb5b93764127b3cc056c4ecba2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.926545 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9eb3bcbece34f14a05dc7d95ca7ece2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.928885 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 841d3a3c98af45339836d9849ff712fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.928897 1850 raft_consensus.cc:491] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.929059 1850 raft_consensus.cc:513] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.930990 2045 raft_consensus.cc:491] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.931138 2045 raft_consensus.cc:513] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.932829 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215021f96ce54429b8aa5e19668fba0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.934439 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7485d0b61b2242e688d4b47cebfcb9cc" } new_table_name: "default.Table_50c078cf2e4148df96289ccba9f2da2d" modify_external_catalogs: true
I20250625 07:52:30.936932 1158 tablet_bootstrap.cc:654] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:30.938023 1162 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:30.938202 1162 ts_tablet_manager.cc:1397] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.003s
I20250625 07:52:30.939029 1162 raft_consensus.cc:357] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:30.939209 1162 raft_consensus.cc:383] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.939306 1162 raft_consensus.cc:738] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.939425 985 leader_election.cc:304] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:30.939695 2028 raft_consensus.cc:2747] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:30.941054 880 tablet_service.cc:1468] Processing CreateTablet for tablet 67d5cb5b93764127b3cc056c4ecba2df (DEFAULT_TABLE table=default.Table_8c9a216aabd74a76990345c02842ea74 [id=f8fb9cf5b8824ec78a5bd07bd13f1428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.941347 890 tablet_service.cc:1468] Processing CreateTablet for tablet f0fe3a2f896c4100a4d484196094f40b (DEFAULT_TABLE table=default.Table_65094aac02f94466a02285f60f743c29 [id=2b8de9c3cc6a4e979eff7f97301b96f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.941481 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e342d843577548959d8be6a80119e769" } new_table_name: "default.Table_3080d01f8b9647b9921a8e13a0b4b9c9" modify_external_catalogs: true
I20250625 07:52:30.942152 894 tablet_service.cc:1468] Processing CreateTablet for tablet 57717427b8bf4024b60cdc341280ac98 (DEFAULT_TABLE table=default.Table_864fa9594e1d4f97aef0122611c515cb [id=5dddb2dfe1c844418d2306ca3904d15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.942381 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8a13381cae324e9494a2497d0190a868 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.942623 891 tablet_service.cc:1468] Processing CreateTablet for tablet 86d9f99c3ed34b2794236282040c2c49 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.947404 888 tablet_service.cc:1468] Processing CreateTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.947412 1163 raft_consensus.cc:357] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.948961 1163 raft_consensus.cc:383] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:30.949023 1163 raft_consensus.cc:738] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:30.949270 1163 consensus_queue.cc:260] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.950548 1163 ts_tablet_manager.cc:1428] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20250625 07:52:30.953919 1166 ts_tablet_manager.cc:1918] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:30.954064 1166 log.cc:1199] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:30.954387 1166 ts_tablet_manager.cc:1939] T cd728ffc2a274deea3fad533d1eaa6da P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:30.956869 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.956885 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 57717427b8bf4024b60cdc341280ac98 (DEFAULT_TABLE table=default.Table_864fa9594e1d4f97aef0122611c515cb [id=5dddb2dfe1c844418d2306ca3904d15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.957271 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d06e9aae7eb408c99d97e52fc9f691c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.957419 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57717427b8bf4024b60cdc341280ac98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.959642 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 86d9f99c3ed34b2794236282040c2c49 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.959892 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad44a68d20774c8daa75b9252f90e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.959976 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d9f99c3ed34b2794236282040c2c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.965634 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_a23788a3c209434db9f0ef44e0220672 [id=c80e78f1ef124af2b8f405e7d162ade8]). Expected version 2 got 0
I20250625 07:52:30.968122 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 8a13381cae324e9494a2497d0190a868 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.968513 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a13381cae324e9494a2497d0190a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.968468 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5009e956d43a46049b8642db6d72fa1b (DEFAULT_TABLE table=default.Table_cb3c32e8a2f246b88ee85462a6d3ff99 [id=398c889dfe0f4b6c9c4e504cc87b6a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.968792 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5009e956d43a46049b8642db6d72fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.968122 1028 tablet_service.cc:1468] Processing CreateTablet for tablet a571983779854eb398b3b0b42c34f83a (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:30.969005 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a571983779854eb398b3b0b42c34f83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.971613 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13b166a67304921b4fdc6ddc68f2035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.972079 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9eb3bcbece34f14a05dc7d95ca7ece2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.973047 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e06a96ef0847eb9568cb363e1ed6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.973505 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c0e148f958453c8c26f8ba6460d13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.973800 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a76704c36b45e2afe54376e30f46fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.974711 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 215021f96ce54429b8aa5e19668fba0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.975162 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67d5cb5b93764127b3cc056c4ecba2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.980409 1793 raft_consensus.cc:491] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:30.980552 1793 raft_consensus.cc:513] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:30.981268 1793 leader_election.cc:290] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:30.981858 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fe3a2f896c4100a4d484196094f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.984906 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57717427b8bf4024b60cdc341280ac98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.986995 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a13381cae324e9494a2497d0190a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.987970 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d9f99c3ed34b2794236282040c2c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.989451 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d06e9aae7eb408c99d97e52fc9f691c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.991679 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 841d3a3c98af45339836d9849ff712fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:30.991880 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ad97f0e793e84237beba40faefc5ec38" } new_table_name: "default.Table_7849c32289204dd5a9024a89fd91292c" modify_external_catalogs: true
I20250625 07:52:31.016216 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 7df088d0ba6f4b1888aa48400a2b7fa3 (DEFAULT_TABLE table=default.Table_033160d4bb6d460093d51e5969e61194 [id=27dd4eccab7640f4a637f7fe850ac5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.016439 1025 tablet_service.cc:1468] Processing CreateTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.016209 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 57a8128b52924e4f9ad8feaa818c9b2f (DEFAULT_TABLE table=default.Table_6c72c34aa1954d40b2be6b4988c05f5b [id=8aaa6a11f2da4307b02dd6fb67bcf457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.016652 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df088d0ba6f4b1888aa48400a2b7fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.016927 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.017228 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a354a11c894b8f992f3d3cdd66cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.017385 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a8128b52924e4f9ad8feaa818c9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.020797 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedc38018eb6412b9e6ca3f0ee34cf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.028507 1166 tablet_replica.cc:331] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:31.016731 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 319c173d175644b2aea6e8ec4060d51a (DEFAULT_TABLE table=default.Table_24fa140ea2d042a8ac731454569eebcb [id=4fe781a5028e476bab66b636b594c7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.028751 1166 raft_consensus.cc:2241] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.028802 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c173d175644b2aea6e8ec4060d51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.028927 1166 raft_consensus.cc:2270] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.032207 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7f926e49307e4d96b5613217b978631e (DEFAULT_TABLE table=default.Table_93934279d504441390d7f1234efdab8e [id=b2d4fe8c1d394f009e453c674bad0aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.032387 2045 leader_election.cc:290] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.032536 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f926e49307e4d96b5613217b978631e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.032536 2045 raft_consensus.cc:491] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.032661 2045 raft_consensus.cc:513] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.033317 2045 leader_election.cc:290] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.033708 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 (DEFAULT_TABLE table=default.Table_4801dc1c88174eb1838ebc1ebbd93d6c [id=2942a8ea319441f0987b73a6db824ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.034083 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05991b31ebf945f29fca8c060b21e7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.034866 1160 tablet_replica.cc:331] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.035104 1160 raft_consensus.cc:2241] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.035277 1160 raft_consensus.cc:2270] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.035851 1160 ts_tablet_manager.cc:1905] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.037698 1162 consensus_queue.cc:260] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.038237 1162 ts_tablet_manager.cc:1428] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.100s user 0.002s sys 0.002s
I20250625 07:52:31.038563 1162 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.040233 1162 tablet_bootstrap.cc:654] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.042589 1850 leader_election.cc:290] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.043504 1166 ts_tablet_manager.cc:1905] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.044925 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.045018 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ab38e0cdcd14f209e6877f2e4a46e98" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.046061 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f4359fd842a4760bae75169e8914f1c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.046173 713 leader_election.cc:343] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:31.046379 713 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.045663 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d109c9ad2e7d452e9533297a490c2031 (DEFAULT_TABLE table=default.Table_b2d591e4706149daa6e50d9466ea91ea [id=426f98767f65431baa762eccfacce5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.046820 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d109c9ad2e7d452e9533297a490c2031. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:31.049427 713 leader_election.cc:343] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.050174 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.051230 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:31.051436 915 raft_consensus.cc:2466] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:31.051502 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f4359fd842a4760bae75169e8914f1c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:31.051661 914 raft_consensus.cc:2466] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:31.052013 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 019959e9abed46fbbbb4755cb01cef44 (table default.Table_2c27be0b9107412facdd1bc4d768eebb [id=ea62e41d8d904cf5ac63b9d0a62cde09]) successfully deleted
I20250625 07:52:31.052575 880 tablet_service.cc:1468] Processing CreateTablet for tablet a571983779854eb398b3b0b42c34f83a (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.052997 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a571983779854eb398b3b0b42c34f83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.053145 893 tablet_service.cc:1468] Processing CreateTablet for tablet 7f926e49307e4d96b5613217b978631e (DEFAULT_TABLE table=default.Table_93934279d504441390d7f1234efdab8e [id=b2d4fe8c1d394f009e453c674bad0aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.054454 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f926e49307e4d96b5613217b978631e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.055634 887 tablet_service.cc:1468] Processing CreateTablet for tablet d109c9ad2e7d452e9533297a490c2031 (DEFAULT_TABLE table=default.Table_b2d591e4706149daa6e50d9466ea91ea [id=426f98767f65431baa762eccfacce5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.043985 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 588a3484db8a48f3b1cee1862505c6a0 (DEFAULT_TABLE table=default.Table_8813030417384a41a121df21b15656e6 [id=05e29cc8f6bc41098b1b62bf3be52733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.056499 886 tablet_service.cc:1468] Processing CreateTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 (DEFAULT_TABLE table=default.Table_4801dc1c88174eb1838ebc1ebbd93d6c [id=2942a8ea319441f0987b73a6db824ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.058012 879 tablet_service.cc:1468] Processing CreateTablet for tablet 319c173d175644b2aea6e8ec4060d51a (DEFAULT_TABLE table=default.Table_24fa140ea2d042a8ac731454569eebcb [id=4fe781a5028e476bab66b636b594c7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.058518 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588a3484db8a48f3b1cee1862505c6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.058542 882 tablet_service.cc:1468] Processing CreateTablet for tablet 588a3484db8a48f3b1cee1862505c6a0 (DEFAULT_TABLE table=default.Table_8813030417384a41a121df21b15656e6 [id=05e29cc8f6bc41098b1b62bf3be52733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.059449 1162 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.059473 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 998) took 1154 ms. Trace:
I20250625 07:52:31.059626 1162 ts_tablet_manager.cc:1397] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250625 07:52:31.059613 1161 rpcz_store.cc:276] 0625 07:52:29.904833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:29.904883 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:30.774327 (+869444us) ts_tablet_manager.cc:1145] Deleting tablet 7ad78944352e4274acd8d9889b1d32b8
0625 07:52:30.774336 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:30.832378 (+ 58042us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:30.835149 (+ 2771us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.059453 (+224304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":869268}
I20250625 07:52:31.060263 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1008) took 1018 ms. Trace:
I20250625 07:52:31.060422 1161 rpcz_store.cc:276] 0625 07:52:30.041566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.041619 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:31.060153 (+1018534us) ts_tablet_manager.cc:1145] Deleting tablet 33f56c267457451e864e277e110c68fa
0625 07:52:31.060162 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.060248 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995282}
I20250625 07:52:31.060446 1162 raft_consensus.cc:357] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.060626 1162 raft_consensus.cc:383] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.060738 1162 raft_consensus.cc:738] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.061066 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1010) took 1013 ms. Trace:
I20250625 07:52:31.061000 1162 consensus_queue.cc:260] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.061197 1161 rpcz_store.cc:276] 0625 07:52:30.047293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.047355 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:31.060962 (+1013607us) ts_tablet_manager.cc:1145] Deleting tablet 883c5592cc014b8eb7d85f62242e5b02
0625 07:52:31.060971 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.061052 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995915}
I20250625 07:52:31.061564 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1011) took 1003 ms. Trace:
I20250625 07:52:31.061694 1161 rpcz_store.cc:276] 0625 07:52:30.057676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.057747 (+ 71us) service_pool.cc:225] Handling call
0625 07:52:31.061468 (+1003721us) ts_tablet_manager.cc:1145] Deleting tablet ac3e3937ef074eb78550f9dd4965c7b7
0625 07:52:31.061478 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.061549 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992686}
I20250625 07:52:31.061829 1162 ts_tablet_manager.cc:1428] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250625 07:52:31.062203 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1012) took 1003 ms. Trace:
I20250625 07:52:31.062364 1162 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.062315 1161 rpcz_store.cc:276] 0625 07:52:30.058813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.058868 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:31.062111 (+1003243us) ts_tablet_manager.cc:1145] Deleting tablet 34a338005a7b44b9b0fba7a7096cfcb8
0625 07:52:31.062119 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.062189 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993291}
I20250625 07:52:31.063042 1161 tablet_replica.cc:331] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.063985 1162 tablet_bootstrap.cc:654] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.065670 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c0cc3ba09d5e4c3d88e74fffc97eea8b
I20250625 07:52:31.066212 594 catalog_manager.cc:6652] ReplaceTablet: tablet c0cc3ba09d5e4c3d88e74fffc97eea8b deleted and replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4
I20250625 07:52:31.069193 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b374bdf422d9423b85441edf89cc3c95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.069806 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b374bdf422d9423b85441edf89cc3c95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.062085 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7ad78944352e4274acd8d9889b1d32b8 (table default.Table_bb291540701c4a4fbb5f3fa31fc4b684 [id=d24d9520ff924056a204dfd910f4dbef]) successfully deleted
I20250625 07:52:31.070556 1163 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:31.072198 1163 tablet_bootstrap.cc:654] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.075143 876 tablet_service.cc:1468] Processing CreateTablet for tablet 5009e956d43a46049b8642db6d72fa1b (DEFAULT_TABLE table=default.Table_cb3c32e8a2f246b88ee85462a6d3ff99 [id=398c889dfe0f4b6c9c4e504cc87b6a4a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:31.070677 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 33f56c267457451e864e277e110c68fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f56c267457451e864e277e110c68fa already in progress: creating tablet
W20250625 07:52:31.076009 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 883c5592cc014b8eb7d85f62242e5b02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 883c5592cc014b8eb7d85f62242e5b02 already in progress: creating tablet
W20250625 07:52:31.077554 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ac3e3937ef074eb78550f9dd4965c7b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac3e3937ef074eb78550f9dd4965c7b7 already in progress: creating tablet
W20250625 07:52:31.077862 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 34a338005a7b44b9b0fba7a7096cfcb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34a338005a7b44b9b0fba7a7096cfcb8 already in progress: creating tablet
W20250625 07:52:31.078616 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 731a30dc035146d584d11cbecd12baf0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 731a30dc035146d584d11cbecd12baf0 already in progress: creating tablet
I20250625 07:52:31.080116 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet cd728ffc2a274deea3fad533d1eaa6da (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:31.081285 755 tablet_service.cc:1468] Processing CreateTablet for tablet f0fe3a2f896c4100a4d484196094f40b (DEFAULT_TABLE table=default.Table_65094aac02f94466a02285f60f743c29 [id=2b8de9c3cc6a4e979eff7f97301b96f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.081730 739 tablet_service.cc:1468] Processing CreateTablet for tablet a571983779854eb398b3b0b42c34f83a (DEFAULT_TABLE table=default.Table_9e0c2aded59d426c9ce157d9d9038958 [id=34e7c2abf6dd44dfa9a7c9f39a31935a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.082480 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5009e956d43a46049b8642db6d72fa1b (DEFAULT_TABLE table=default.Table_cb3c32e8a2f246b88ee85462a6d3ff99 [id=398c889dfe0f4b6c9c4e504cc87b6a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.082947 757 tablet_service.cc:1468] Processing CreateTablet for tablet 86d9f99c3ed34b2794236282040c2c49 (DEFAULT_TABLE table=default.Table_4c1dbe5c89ac43e59af58d81b811f786 [id=c119b9683a334e5185c5902d15f52ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.083144 748 tablet_service.cc:1468] Processing CreateTablet for tablet d109c9ad2e7d452e9533297a490c2031 (DEFAULT_TABLE table=default.Table_b2d591e4706149daa6e50d9466ea91ea [id=426f98767f65431baa762eccfacce5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.083606 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c (DEFAULT_TABLE table=default.Table_144dd070706a4d6e8d042f25ec83125e [id=b9f6e96dc7814b31885eff59052de9e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.083802 758 tablet_service.cc:1468] Processing CreateTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 (DEFAULT_TABLE table=default.Table_4801dc1c88174eb1838ebc1ebbd93d6c [id=2942a8ea319441f0987b73a6db824ff1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.083959 747 tablet_service.cc:1468] Processing CreateTablet for tablet 588a3484db8a48f3b1cee1862505c6a0 (DEFAULT_TABLE table=default.Table_8813030417384a41a121df21b15656e6 [id=05e29cc8f6bc41098b1b62bf3be52733]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.084187 751 tablet_service.cc:1468] Processing CreateTablet for tablet 319c173d175644b2aea6e8ec4060d51a (DEFAULT_TABLE table=default.Table_24fa140ea2d042a8ac731454569eebcb [id=4fe781a5028e476bab66b636b594c7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.084342 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7f926e49307e4d96b5613217b978631e (DEFAULT_TABLE table=default.Table_93934279d504441390d7f1234efdab8e [id=b2d4fe8c1d394f009e453c674bad0aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.084465 1850 raft_consensus.cc:491] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.084636 1850 raft_consensus.cc:513] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.084833 752 tablet_service.cc:1468] Processing CreateTablet for tablet 57a8128b52924e4f9ad8feaa818c9b2f (DEFAULT_TABLE table=default.Table_6c72c34aa1954d40b2be6b4988c05f5b [id=8aaa6a11f2da4307b02dd6fb67bcf457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.085060 745 tablet_service.cc:1468] Processing CreateTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.085186 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7df088d0ba6f4b1888aa48400a2b7fa3 (DEFAULT_TABLE table=default.Table_033160d4bb6d460093d51e5969e61194 [id=27dd4eccab7640f4a637f7fe850ac5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.085577 1850 leader_election.cc:290] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.085626 744 tablet_service.cc:1468] Processing CreateTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.085711 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b with delete_type TABLET_DATA_DELETED (replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.086090 743 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.086836 746 tablet_service.cc:1468] Processing CreateTablet for tablet 8a13381cae324e9494a2497d0190a868 (DEFAULT_TABLE table=default.Table_9b46e6d83dcd438fb7a25fb360da7435 [id=2f03560d64794b66926f8a4e5e593320]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.087139 753 tablet_service.cc:1468] Processing CreateTablet for tablet 57717427b8bf4024b60cdc341280ac98 (DEFAULT_TABLE table=default.Table_864fa9594e1d4f97aef0122611c515cb [id=5dddb2dfe1c844418d2306ca3904d15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.094341 877 tablet_service.cc:1468] Processing CreateTablet for tablet 57a8128b52924e4f9ad8feaa818c9b2f (DEFAULT_TABLE table=default.Table_6c72c34aa1954d40b2be6b4988c05f5b [id=8aaa6a11f2da4307b02dd6fb67bcf457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.095741 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7df088d0ba6f4b1888aa48400a2b7fa3 (DEFAULT_TABLE table=default.Table_033160d4bb6d460093d51e5969e61194 [id=27dd4eccab7640f4a637f7fe850ac5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.095885 885 tablet_service.cc:1468] Processing CreateTablet for tablet bedc38018eb6412b9e6ca3f0ee34cf87 (DEFAULT_TABLE table=default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.096804 891 tablet_service.cc:1468] Processing CreateTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.097143 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b with delete_type TABLET_DATA_DELETED (replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:31.097317 850 leader_election.cc:343] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:31.099609 713 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:31.099892 713 leader_election.cc:304] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.100291 1850 raft_consensus.cc:2747] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.100531 713 leader_election.cc:304] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.100981 1850 raft_consensus.cc:2802] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:31.101173 1850 raft_consensus.cc:491] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:31.101374 1850 raft_consensus.cc:3058] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:31.101465 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d0a75906385744e5ba6938f451a2d7ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a75906385744e5ba6938f451a2d7ec already in progress: creating tablet
I20250625 07:52:31.105937 2096 raft_consensus.cc:491] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.106251 2096 raft_consensus.cc:513] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.108031 2096 leader_election.cc:290] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.109253 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:31.110594 851 leader_election.cc:343] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.110734 851 leader_election.cc:304] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.111218 1793 raft_consensus.cc:2747] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.087469 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b9491eb269948a29888a7301ea75d42" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.088378 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b with delete_type TABLET_DATA_DELETED (replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.117913 1161 raft_consensus.cc:2241] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.118144 1161 raft_consensus.cc:2270] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.101182 713 leader_election.cc:304] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:31.121711 713 raft_consensus.cc:2684] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250625 07:52:31.126050 713 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3b9491eb269948a29888a7301ea75d42
I20250625 07:52:31.128186 1160 ts_tablet_manager.cc:1918] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.128309 1160 log.cc:1199] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:31.128616 1160 ts_tablet_manager.cc:1939] T b2bb060cdd9a48688967989d7491cfc1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:31.128918 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df088d0ba6f4b1888aa48400a2b7fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.129451 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a354a11c894b8f992f3d3cdd66cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.131855 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a8128b52924e4f9ad8feaa818c9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.132405 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedc38018eb6412b9e6ca3f0ee34cf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.133417 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c173d175644b2aea6e8ec4060d51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.134908 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d109c9ad2e7d452e9533297a490c2031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.135277 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588a3484db8a48f3b1cee1862505c6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.135695 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05991b31ebf945f29fca8c060b21e7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.136343 1793 raft_consensus.cc:491] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.136474 1793 raft_consensus.cc:513] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.137182 1793 leader_election.cc:290] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.139214 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f04016cda7f746d49e30be3b1c9c18b6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.139943 850 leader_election.cc:343] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.143957 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5009e956d43a46049b8642db6d72fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.149291 1160 tablet_replica.cc:331] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.149654 1160 raft_consensus.cc:2241] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.150071 1160 raft_consensus.cc:2270] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.151062 1160 ts_tablet_manager.cc:1905] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.152714 1793 raft_consensus.cc:491] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.152859 1793 raft_consensus.cc:513] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.153517 1793 leader_election.cc:290] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.155622 1793 raft_consensus.cc:491] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.155759 1793 raft_consensus.cc:513] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.156343 1793 leader_election.cc:290] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.130016 1850 raft_consensus.cc:513] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.131101 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a571983779854eb398b3b0b42c34f83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.138175 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f04016cda7f746d49e30be3b1c9c18b6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:31.174638 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe5c37fc9bd49c685de8f469a7af0c6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:31.175257 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d393a85f5fde4b3fb57e5b3ec372791c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:31.181591 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 54606us
I20250625 07:52:31.181942 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5009e956d43a46049b8642db6d72fa1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.182087 1850 leader_election.cc:290] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.182216 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588a3484db8a48f3b1cee1862505c6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.182515 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a8128b52924e4f9ad8feaa818c9b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.182755 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05991b31ebf945f29fca8c060b21e7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.186837 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c173d175644b2aea6e8ec4060d51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.187248 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e60d4f9aed44082b9905fbb44ffb110" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.187830 713 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.188059 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f926e49307e4d96b5613217b978631e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.188609 713 leader_election.cc:304] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:31.190109 713 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.190862 2045 raft_consensus.cc:2747] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.191069 2045 raft_consensus.cc:491] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.191326 2045 raft_consensus.cc:513] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.192210 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:31.192663 2045 leader_election.cc:290] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:31.192899 713 leader_election.cc:343] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.193324 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7480c07af7ce4a3889bd8d42d7419d57" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.194341 713 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.195148 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df088d0ba6f4b1888aa48400a2b7fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.195364 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a354a11c894b8f992f3d3cdd66cc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.195613 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d109c9ad2e7d452e9533297a490c2031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.195972 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedc38018eb6412b9e6ca3f0ee34cf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.196484 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a13381cae324e9494a2497d0190a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.198474 1161 ts_tablet_manager.cc:1905] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.199885 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d06e9aae7eb408c99d97e52fc9f691c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.200349 1166 ts_tablet_manager.cc:1918] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.200531 1166 log.cc:1199] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9f4359fd842a4760bae75169e8914f1c
I20250625 07:52:31.201040 1166 ts_tablet_manager.cc:1939] T 9f4359fd842a4760bae75169e8914f1c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.204758 1162 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.204975 1162 ts_tablet_manager.cc:1397] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.143s user 0.028s sys 0.035s
I20250625 07:52:31.205338 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86d9f99c3ed34b2794236282040c2c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.210378 1158 tablet_bootstrap.cc:492] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.210532 1158 ts_tablet_manager.cc:1397] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.289s user 0.009s sys 0.021s
I20250625 07:52:31.211287 1158 raft_consensus.cc:357] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.211467 1158 raft_consensus.cc:383] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.211524 1158 raft_consensus.cc:738] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.211740 1158 consensus_queue.cc:260] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.212224 1158 ts_tablet_manager.cc:1428] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:31.212512 1158 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.213290 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9f4359fd842a4760bae75169e8914f1c (table default.Table_56a7bd0b475e451b8afa385ff9915ba5 [id=914805a4f6684943a2e76c948377f326]) successfully deleted
W20250625 07:52:31.214170 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a899790713d4646a850971bc3aebd9f consensus state has no leader
I20250625 07:52:31.214849 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_143a766c3eb149338d9b90575b6deaf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.215435 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_143a766c3eb149338d9b90575b6deaf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.215673 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0b1f254b07ab43f082eb0d6b1b02c48c" } new_table_name: "default.Table_47ab6034ce084c5d8e41812985ab3aea" modify_external_catalogs: true
I20250625 07:52:31.216001 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57717427b8bf4024b60cdc341280ac98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.216233 1166 tablet_replica.cc:331] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
W20250625 07:52:31.216962 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f9955688a574c80a8b14c2018ed41f7 consensus state has no leader
I20250625 07:52:31.217254 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ad8d548753e44f4ea9c2109f537c6194" } modify_external_catalogs: true
I20250625 07:52:31.217427 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ad8d548753e44f4ea9c2109f537c6194" } modify_external_catalogs: true
I20250625 07:52:31.218567 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:31.219269 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_21de3cd434744102b0d5ffbbc17c18c3" } modify_external_catalogs: true
I20250625 07:52:31.219410 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_21de3cd434744102b0d5ffbbc17c18c3" } modify_external_catalogs: true
W20250625 07:52:31.219924 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b83f4c8544946d9adb3346df200b215 consensus state has no leader
I20250625 07:52:31.220217 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4aac2b881bfb4ada8db2169e25c54197" } modify_external_catalogs: true
I20250625 07:52:31.220302 2028 raft_consensus.cc:491] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.220355 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4aac2b881bfb4ada8db2169e25c54197" } modify_external_catalogs: true
I20250625 07:52:31.220432 2028 raft_consensus.cc:513] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.220834 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b2bb060cdd9a48688967989d7491cfc1 (table default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]) successfully deleted
I20250625 07:52:31.221076 2028 leader_election.cc:290] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:31.221100 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d0a75906385744e5ba6938f451a2d7ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a75906385744e5ba6938f451a2d7ec already in progress: creating tablet
I20250625 07:52:31.221431 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f30892ec994d427591925e7f321f3a4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:31.221519 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe5c37fc9bd49c685de8f469a7af0c6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.221988 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f30892ec994d427591925e7f321f3a4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.231194 985 leader_election.cc:343] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 802c22a6346442c88abbeb4b9ef20c57
I20250625 07:52:31.231393 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_57bbb43218994668b3c285c21c088ffa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.231966 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57bbb43218994668b3c285c21c088ffa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.233449 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_34ac7b0478f94f3d8a0c39d6671c95e8" } new_table_name: "default.Table_5dc84a68d54248fe8ac0f3b4453374e0" modify_external_catalogs: true
W20250625 07:52:31.236608 851 leader_election.cc:343] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.236756 851 leader_election.cc:304] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.237147 1793 raft_consensus.cc:2747] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:31.237527 851 leader_election.cc:343] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:31.237870 851 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.238727 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d393a85f5fde4b3fb57e5b3ec372791c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.250289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5986bb52335c46bf845c48cb044495b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5986bb52335c46bf845c48cb044495b1 already in progress: creating tablet
I20250625 07:52:31.251027 1158 tablet_bootstrap.cc:654] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:31.251317 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7ef2eb85358b414c9755f81f97beaddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef2eb85358b414c9755f81f97beaddb already in progress: creating tablet
W20250625 07:52:31.252351 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a already in progress: creating tablet
I20250625 07:52:31.252528 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fe3a2f896c4100a4d484196094f40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.258591 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_a23788a3c209434db9f0ef44e0220672 [id=c80e78f1ef124af2b8f405e7d162ade8]). Expected version 2 got 0
I20250625 07:52:31.261865 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ecd308318d99472b929b8e811d4d613f" } modify_external_catalogs: true
I20250625 07:52:31.262117 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ecd308318d99472b929b8e811d4d613f" } modify_external_catalogs: true
W20250625 07:52:31.263481 986 leader_election.cc:343] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 802c22a6346442c88abbeb4b9ef20c57
I20250625 07:52:31.263639 986 leader_election.cc:304] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.264189 2028 raft_consensus.cc:2747] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.267853 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e26b4b26dfe434ba65119e99f41bb8c" } modify_external_catalogs: true
I20250625 07:52:31.268049 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e26b4b26dfe434ba65119e99f41bb8c" } modify_external_catalogs: true
I20250625 07:52:31.272922 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 10c7f4b4187c428bbec136c85219da4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.272972 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c5c50eb840b844438807b8cb32015ac8 (DEFAULT_TABLE table=default.Table_c434f9d332484fc09394c912e328bf47 [id=bb39acc0e66a4baf9d800fe0663da1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.278407 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 868106f037d2484e94eab2f0d6394f92 (DEFAULT_TABLE table=default.Table_d45cd26e5aab475199c9c4b5ffb4d900 [id=47ccb825fdc64005a9b972d0d4c83fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.279304 1020 tablet_service.cc:1468] Processing CreateTablet for tablet a5c46a9100514d15a30be2d918da556f (DEFAULT_TABLE table=default.Table_0b1f254b07ab43f082eb0d6b1b02c48c [id=7c645e964e0a4133bfd01dc186477fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.280102 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 83ada5281a2e4bf682c889778b478815 (DEFAULT_TABLE table=default.Table_22d71b8b5d5442c2b676651f64104641 [id=e321c807f6d24001a155468ad13b0851]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.280246 1166 raft_consensus.cc:2241] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.280423 1166 raft_consensus.cc:2270] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.280771 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 95c00713052447e0bc2d1aa51f0e1a1f (DEFAULT_TABLE table=default.Table_b7c1b9f303dd44bda543ea0259d42097 [id=e93e47df027e4eb29cdc8b6bb4f3f26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.281003 1166 ts_tablet_manager.cc:1905] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.283110 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 27e94c4fe10d48368a212eac8d8d09aa (DEFAULT_TABLE table=default.Table_b8146a0332fc4ef897cbf4a745c83b5f [id=846cd607927e405aadbaca92e012c507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.283712 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d320734990404d1e9222bea1d88a18fe (DEFAULT_TABLE table=default.Table_b1e0cba0d5f340a88739f9994705b2e7 [id=c67fc2b8004245ba93a726d85b0a8d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.284327 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 08521c443c284f6aac4342b8ceb24d86 (DEFAULT_TABLE table=default.Table_b649eef37b514286a98a4bc732c1870a [id=8e8a5851c650460b9162c7edf8682a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.284657 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b3c54204d5446c688cb123edbd430c1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:31.284837 915 raft_consensus.cc:3058] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:31.287974 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e (DEFAULT_TABLE table=default.Table_2e4dcd29c1264e7ead844712846cd17a [id=a4b8b5f15118470f8d0868221c941e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.289327 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 (DEFAULT_TABLE table=default.Table_e046fc4755bc40769ba73e88edfe6b53 [id=24c507fc082a44ef8e0e330294916b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.299944 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 76ed33a26e764dba80b601edf5454600 (DEFAULT_TABLE table=default.Table_37ecb82090564313862912776a682691 [id=8306fec40b4648cb9aaf36094ff5d69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.300926 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 1748e9e9bdbc453da0c3ff120ae70ffd (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.301123 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ed33a26e764dba80b601edf5454600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.301282 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748e9e9bdbc453da0c3ff120ae70ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.301411 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 5d251926c3c04d45b103b59b4c74db23 (DEFAULT_TABLE table=default.Table_8a81d60515d647b48496abdf8ef41532 [id=06374f1dde324a0ba96c7bb6c56de7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.301723 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c50eb840b844438807b8cb32015ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.301719 1793 raft_consensus.cc:491] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.301885 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d320734990404d1e9222bea1d88a18fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.301851 1793 raft_consensus.cc:513] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.302206 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e94c4fe10d48368a212eac8d8d09aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.302570 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ada5281a2e4bf682c889778b478815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.302923 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08521c443c284f6aac4342b8ceb24d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.303304 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c00713052447e0bc2d1aa51f0e1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.303804 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e538d6dd213f4b66870d314dadbf5c9a (DEFAULT_TABLE table=default.Table_342146f029294bdaa3d7e68bc9e8d6b2 [id=33863857a134475a8f0b0d04a64c2836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.304243 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 09120e19ea0f4ec7b94c89d173d3609f (DEFAULT_TABLE table=default.Table_bb75ba8ad3674db084f085ca2c573442 [id=f0bc9284b37442faac1f5f0e4faea800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.304515 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.304584 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 36c61f71a1674957805a48ccd6d0d4b3 (DEFAULT_TABLE table=default.Table_86b56ad960e1457ca74b3db973f6cbd0 [id=a7da47d868124bfbbf316a81083f438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.304708 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 363fee07295c44139fd713b33c82623c (DEFAULT_TABLE table=default.Table_2f21fd181ae64529a96b8dd7b83c8cad [id=c1e08f7897eb49a3a0d6c4b257e1a5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.304941 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.305369 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c46a9100514d15a30be2d918da556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.305922 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868106f037d2484e94eab2f0d6394f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.308647 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.313380 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 184cd64146f141db8e870af121a7772f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:31.313848 593 master_service.cc:948] ReplaceTablet: received request to replace tablet fd9677bf99f041c2ade65c94f0170609 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:31.314350 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a59e4843e6594fc8bb82fe450bb820cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.314872 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a59e4843e6594fc8bb82fe450bb820cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.318024 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd (DEFAULT_TABLE table=default.Table_4a44da8c8ff346118e917baceb47de8e [id=8f05b151cc6d4b6ba0c5670a94a3e8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.318411 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c51c79ac41cebfca7dcc567ddecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.321290 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_00236ecf6f4146f2be402a5a04cb0d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.321969 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00236ecf6f4146f2be402a5a04cb0d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.323359 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81d13afd21f34e8794b603ccaa0b1d84 consensus state has no leader
I20250625 07:52:31.323707 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_40f5f254457348eeb98731f1bab610f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.324182 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40f5f254457348eeb98731f1bab610f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.324951 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5c46a9100514d15a30be2d918da556f consensus state has no leader
I20250625 07:52:31.325843 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7c8abe36eab5498eb9bc3a0355101dd6
I20250625 07:52:31.326507 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 08cd482ceec244c2bed2b116d5c14d89
I20250625 07:52:31.327257 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 308b387d366c4d7884898937a9fc7690
I20250625 07:52:31.330996 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c61f71a1674957805a48ccd6d0d4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.332850 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fee07295c44139fd713b33c82623c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.333352 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d251926c3c04d45b103b59b4c74db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.333763 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09120e19ea0f4ec7b94c89d173d3609f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.334208 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e538d6dd213f4b66870d314dadbf5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.337069 2110 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:31.346848 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d92c6481d56745f9a67a5c0cdf374c9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.347501 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d92c6481d56745f9a67a5c0cdf374c9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.350162 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.350363 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 308b387d366c4d7884898937a9fc7690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.350512 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.350689 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 54c2a9e362a6432880f30fd9e8719443 (DEFAULT_TABLE table=default.Table_b4fdd7ed1d8d4f2d814d8e22fea75b12 [id=c02ad9604e7240858e54579966bede5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.350734 1014 tablet_service.cc:1468] Processing CreateTablet for tablet b70ba88bd1f84247b9e18659d7112cc8 (DEFAULT_TABLE table=default.Table_30292e2cad9c4c039fce627374179f83 [id=6adf8fb3937741fbb867968ef8f6197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.351724 1158 tablet_bootstrap.cc:492] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.351724 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 484a49fab6134d059c64f5a3f2b29e03
I20250625 07:52:31.351855 1158 ts_tablet_manager.cc:1397] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.139s user 0.005s sys 0.001s
I20250625 07:52:31.352658 1158 raft_consensus.cc:357] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.352850 1158 raft_consensus.cc:383] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.352916 1158 raft_consensus.cc:738] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.353107 1158 consensus_queue.cc:260] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.353653 1158 ts_tablet_manager.cc:1428] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:31.353899 1158 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.355317 1158 tablet_bootstrap.cc:654] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.356508 1793 leader_election.cc:290] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.357501 1793 raft_consensus.cc:491] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.357606 1793 raft_consensus.cc:513] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.358165 1793 leader_election.cc:290] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.361582 2103 raft_consensus.cc:491] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.361737 2103 raft_consensus.cc:513] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.362371 2103 leader_election.cc:290] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.364109 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bb74391a1c4e2792226d259a07f977" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.364462 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "547b758c2e6c4558bb232f14b37aa81a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.367303 893 tablet_service.cc:1468] Processing CreateTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd (DEFAULT_TABLE table=default.Table_4a44da8c8ff346118e917baceb47de8e [id=8f05b151cc6d4b6ba0c5670a94a3e8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.371838 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.372264 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef8838908e649d995e0131de3130dea (DEFAULT_TABLE table=default.Table_b333f5398ea94718a9f9c3208775b51d [id=8c823b8f9cba4a33af9c4c42bb0a0018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.373608 887 tablet_service.cc:1468] Processing CreateTablet for tablet c5c50eb840b844438807b8cb32015ac8 (DEFAULT_TABLE table=default.Table_c434f9d332484fc09394c912e328bf47 [id=bb39acc0e66a4baf9d800fe0663da1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.375908 1158 tablet_bootstrap.cc:492] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.376031 1158 ts_tablet_manager.cc:1397] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250625 07:52:31.376864 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7be8ab5140aa4d48be978519cb99ed31
I20250625 07:52:31.376818 1158 raft_consensus.cc:357] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.376993 1158 raft_consensus.cc:383] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.377048 1158 raft_consensus.cc:738] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.377246 1158 consensus_queue.cc:260] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.377709 1158 ts_tablet_manager.cc:1428] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:31.385185 1163 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:31.385357 1163 ts_tablet_manager.cc:1397] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.315s user 0.010s sys 0.008s
I20250625 07:52:31.386427 886 tablet_service.cc:1468] Processing CreateTablet for tablet 868106f037d2484e94eab2f0d6394f92 (DEFAULT_TABLE table=default.Table_d45cd26e5aab475199c9c4b5ffb4d900 [id=47ccb825fdc64005a9b972d0d4c83fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.389372 1158 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.366493 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 82d661545e924905afa3d08eccac2e65 (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.391389 1158 tablet_bootstrap.cc:654] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.392432 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd (DEFAULT_TABLE table=default.Table_4a44da8c8ff346118e917baceb47de8e [id=8f05b151cc6d4b6ba0c5670a94a3e8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.401432 2028 raft_consensus.cc:491] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.401594 2028 raft_consensus.cc:513] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.402063 1161 ts_tablet_manager.cc:1918] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.402164 1161 log.cc:1199] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a3670cadfa2841949705c65c9b6c0927
I20250625 07:52:31.402283 2028 leader_election.cc:290] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.402464 1161 ts_tablet_manager.cc:1939] T a3670cadfa2841949705c65c9b6c0927 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.405143 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "547b758c2e6c4558bb232f14b37aa81a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:31.408028 750 tablet_service.cc:1468] Processing CreateTablet for tablet c5c50eb840b844438807b8cb32015ac8 (DEFAULT_TABLE table=default.Table_c434f9d332484fc09394c912e328bf47 [id=bb39acc0e66a4baf9d800fe0663da1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.408284 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bb74391a1c4e2792226d259a07f977" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:31.408759 1162 raft_consensus.cc:357] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.408918 1162 raft_consensus.cc:383] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.408975 1162 raft_consensus.cc:738] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:31.409024 851 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:31.409201 851 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.409142 1162 consensus_queue.cc:260] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.409579 1162 ts_tablet_manager.cc:1428] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.204s user 0.001s sys 0.000s
W20250625 07:52:31.412019 986 leader_election.cc:343] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:31.417052 757 tablet_service.cc:1468] Processing CreateTablet for tablet 868106f037d2484e94eab2f0d6394f92 (DEFAULT_TABLE table=default.Table_d45cd26e5aab475199c9c4b5ffb4d900 [id=47ccb825fdc64005a9b972d0d4c83fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.417426 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868106f037d2484e94eab2f0d6394f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.417451 752 tablet_service.cc:1468] Processing CreateTablet for tablet a5c46a9100514d15a30be2d918da556f (DEFAULT_TABLE table=default.Table_0b1f254b07ab43f082eb0d6b1b02c48c [id=7c645e964e0a4133bfd01dc186477fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.417717 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c46a9100514d15a30be2d918da556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.418363 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c51c79ac41cebfca7dcc567ddecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.420238 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c50eb840b844438807b8cb32015ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.420362 1162 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.421058 885 tablet_service.cc:1468] Processing CreateTablet for tablet a5c46a9100514d15a30be2d918da556f (DEFAULT_TABLE table=default.Table_0b1f254b07ab43f082eb0d6b1b02c48c [id=7c645e964e0a4133bfd01dc186477fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.421576 753 tablet_service.cc:1468] Processing CreateTablet for tablet 83ada5281a2e4bf682c889778b478815 (DEFAULT_TABLE table=default.Table_22d71b8b5d5442c2b676651f64104641 [id=e321c807f6d24001a155468ad13b0851]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.421963 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ada5281a2e4bf682c889778b478815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.424876 740 tablet_service.cc:1468] Processing CreateTablet for tablet d320734990404d1e9222bea1d88a18fe (DEFAULT_TABLE table=default.Table_b1e0cba0d5f340a88739f9994705b2e7 [id=c67fc2b8004245ba93a726d85b0a8d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.425159 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d320734990404d1e9222bea1d88a18fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.424592 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e (DEFAULT_TABLE table=default.Table_2e4dcd29c1264e7ead844712846cd17a [id=a4b8b5f15118470f8d0868221c941e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.425628 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.424572 751 tablet_service.cc:1468] Processing CreateTablet for tablet 27e94c4fe10d48368a212eac8d8d09aa (DEFAULT_TABLE table=default.Table_b8146a0332fc4ef897cbf4a745c83b5f [id=846cd607927e405aadbaca92e012c507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.427856 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e94c4fe10d48368a212eac8d8d09aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.427883 2108 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:31.428457 756 tablet_service.cc:1468] Processing CreateTablet for tablet 08521c443c284f6aac4342b8ceb24d86 (DEFAULT_TABLE table=default.Table_b649eef37b514286a98a4bc732c1870a [id=8e8a5851c650460b9162c7edf8682a80]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:31.430841 713 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:31.430954 713 leader_election.cc:304] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.431416 2103 raft_consensus.cc:2747] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.431778 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 (DEFAULT_TABLE table=default.Table_e046fc4755bc40769ba73e88edfe6b53 [id=24c507fc082a44ef8e0e330294916b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.432117 739 tablet_service.cc:1468] Processing CreateTablet for tablet 76ed33a26e764dba80b601edf5454600 (DEFAULT_TABLE table=default.Table_37ecb82090564313862912776a682691 [id=8306fec40b4648cb9aaf36094ff5d69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.432304 743 tablet_service.cc:1468] Processing CreateTablet for tablet e538d6dd213f4b66870d314dadbf5c9a (DEFAULT_TABLE table=default.Table_342146f029294bdaa3d7e68bc9e8d6b2 [id=33863857a134475a8f0b0d04a64c2836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.432125 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5d251926c3c04d45b103b59b4c74db23 (DEFAULT_TABLE table=default.Table_8a81d60515d647b48496abdf8ef41532 [id=06374f1dde324a0ba96c7bb6c56de7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.432513 754 tablet_service.cc:1468] Processing CreateTablet for tablet 36c61f71a1674957805a48ccd6d0d4b3 (DEFAULT_TABLE table=default.Table_86b56ad960e1457ca74b3db973f6cbd0 [id=a7da47d868124bfbbf316a81083f438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.432945 758 tablet_service.cc:1468] Processing CreateTablet for tablet 1748e9e9bdbc453da0c3ff120ae70ffd (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.433277 742 tablet_service.cc:1468] Processing CreateTablet for tablet 09120e19ea0f4ec7b94c89d173d3609f (DEFAULT_TABLE table=default.Table_bb75ba8ad3674db084f085ca2c573442 [id=f0bc9284b37442faac1f5f0e4faea800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.433862 745 tablet_service.cc:1468] Processing CreateTablet for tablet 95c00713052447e0bc2d1aa51f0e1a1f (DEFAULT_TABLE table=default.Table_b7c1b9f303dd44bda543ea0259d42097 [id=e93e47df027e4eb29cdc8b6bb4f3f26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.436818 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.438843 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08521c443c284f6aac4342b8ceb24d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.439877 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70ba88bd1f84247b9e18659d7112cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.442421 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2a9e362a6432880f30fd9e8719443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.444675 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7057fefe95d146c3a4dcf7de25f2d500" } new_table_name: "default.Table_95a6076a07824d0fa2ca5eab23b3bda2" modify_external_catalogs: true
I20250625 07:52:31.445690 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.446442 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b252524928e4ed3a800c849b1ffd8e0" } new_table_name: "default.Table_5428a0666d7a4a21be162dd87b4348d8" modify_external_catalogs: true
I20250625 07:52:31.446933 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.447278 755 tablet_service.cc:1468] Processing CreateTablet for tablet 363fee07295c44139fd713b33c82623c (DEFAULT_TABLE table=default.Table_2f21fd181ae64529a96b8dd7b83c8cad [id=c1e08f7897eb49a3a0d6c4b257e1a5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.447654 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e538d6dd213f4b66870d314dadbf5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.447971 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8f19599ce5b7472c9e5c611e68a3c16b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.446691 713 leader_election.cc:343] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.448748 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_006e977b28ff4dbe9f2091ba6f1879e2" } new_table_name: "default.Table_755e5d8d2b6c4c32bb20ec7903fab9c6" modify_external_catalogs: true
I20250625 07:52:31.449260 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d251926c3c04d45b103b59b4c74db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.449415 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c61f71a1674957805a48ccd6d0d4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.449683 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09120e19ea0f4ec7b94c89d173d3609f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.449962 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.449946 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c6b96e7894e34dc3af94bd1f0e575a8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:31.450139 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 31942f0998bb4118a91f476325851d5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.450276 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748e9e9bdbc453da0c3ff120ae70ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.450399 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c00713052447e0bc2d1aa51f0e1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.452911 1047 raft_consensus.cc:3058] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:31.452983 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f19599ce5b7472c9e5c611e68a3c16b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.454829 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6b96e7894e34dc3af94bd1f0e575a8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.457074 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fee07295c44139fd713b33c82623c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.465456 2028 raft_consensus.cc:491] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.465591 2028 raft_consensus.cc:513] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.466215 2028 leader_election.cc:290] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.470402 2028 raft_consensus.cc:491] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.470554 2028 raft_consensus.cc:513] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.471124 2028 leader_election.cc:290] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.471436 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ed33a26e764dba80b601edf5454600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.472066 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef8838908e649d995e0131de3130dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.473361 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.473984 2110 tablet.cc:1722] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:31.476409 1166 ts_tablet_manager.cc:1918] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.476503 1166 log.cc:1199] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:31.478396 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 308b387d366c4d7884898937a9fc7690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.478878 1161 tablet_replica.cc:331] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.479036 1161 raft_consensus.cc:2241] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.479161 1161 raft_consensus.cc:2270] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.479610 1161 ts_tablet_manager.cc:1905] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.481360 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.483049 749 tablet_service.cc:1468] Processing CreateTablet for tablet 54c2a9e362a6432880f30fd9e8719443 (DEFAULT_TABLE table=default.Table_b4fdd7ed1d8d4f2d814d8e22fea75b12 [id=c02ad9604e7240858e54579966bede5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.485808 1161 ts_tablet_manager.cc:1918] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.485898 1161 log.cc:1199] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7f196473315441c6bf8b5173ed46f833
I20250625 07:52:31.486150 1161 ts_tablet_manager.cc:1939] T 7f196473315441c6bf8b5173ed46f833 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.486135 748 tablet_service.cc:1468] Processing CreateTablet for tablet 82d661545e924905afa3d08eccac2e65 (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.488646 1166 ts_tablet_manager.cc:1939] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.490448 752 tablet_service.cc:1468] Processing CreateTablet for tablet b70ba88bd1f84247b9e18659d7112cc8 (DEFAULT_TABLE table=default.Table_30292e2cad9c4c039fce627374179f83 [id=6adf8fb3937741fbb867968ef8f6197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.490535 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef8838908e649d995e0131de3130dea (DEFAULT_TABLE table=default.Table_b333f5398ea94718a9f9c3208775b51d [id=8c823b8f9cba4a33af9c4c42bb0a0018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.490962 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d6fdf8a3b2db409cb38d372aab234798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.491529 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6fdf8a3b2db409cb38d372aab234798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.491832 986 leader_election.cc:343] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a7cb2bcdd07c42e680eb4f155212423f
I20250625 07:52:31.493534 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 64ca83369ffe46ea9bd756f20da936cf from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:31.496052 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a3670cadfa2841949705c65c9b6c0927 (table default.Table_a881661cd92a4541a73b939c4532acc7 [id=dd2e80e6531a4e3f9be3b2fc19aec035]) successfully deleted
W20250625 07:52:31.503242 985 leader_election.cc:343] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:31.503396 985 leader_election.cc:304] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.503605 2028 raft_consensus.cc:2747] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.504050 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5c50eb840b844438807b8cb32015ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.504297 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4718c51c79ac41cebfca7dcc567ddecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.504539 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 868106f037d2484e94eab2f0d6394f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.505227 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c46a9100514d15a30be2d918da556f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.507870 891 tablet_service.cc:1468] Processing CreateTablet for tablet 95c00713052447e0bc2d1aa51f0e1a1f (DEFAULT_TABLE table=default.Table_b7c1b9f303dd44bda543ea0259d42097 [id=e93e47df027e4eb29cdc8b6bb4f3f26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.508244 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c00713052447e0bc2d1aa51f0e1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.510848 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1016) took 1008 ms. Trace:
I20250625 07:52:31.510951 1161 rpcz_store.cc:276] 0625 07:52:30.502005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.502322 (+ 317us) service_pool.cc:225] Handling call
0625 07:52:31.478832 (+976510us) ts_tablet_manager.cc:1145] Deleting tablet 7f196473315441c6bf8b5173ed46f833
0625 07:52:31.478839 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.482970 (+ 4131us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.485738 (+ 2768us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.510831 (+ 25093us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24353,"tablet-delete.queue_time_us":976316}
I20250625 07:52:31.511238 1161 tablet_replica.cc:331] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.511402 1161 raft_consensus.cc:2241] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.511547 1161 raft_consensus.cc:2270] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.512079 1161 ts_tablet_manager.cc:1905] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.512784 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d661545e924905afa3d08eccac2e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.513288 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:31.519068 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:31.520488 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c744fef0e24cb39c4bb0787adb1cbc" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:31.520596 985 leader_election.cc:343] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a7cb2bcdd07c42e680eb4f155212423f
I20250625 07:52:31.520746 985 leader_election.cc:304] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.520977 2028 raft_consensus.cc:2747] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.521399 876 tablet_service.cc:1468] Processing CreateTablet for tablet 83ada5281a2e4bf682c889778b478815 (DEFAULT_TABLE table=default.Table_22d71b8b5d5442c2b676651f64104641 [id=e321c807f6d24001a155468ad13b0851]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.521481 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.522747 1162 tablet_bootstrap.cc:654] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.523253 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2a9e362a6432880f30fd9e8719443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.524025 894 tablet_service.cc:1468] Processing CreateTablet for tablet d320734990404d1e9222bea1d88a18fe (DEFAULT_TABLE table=default.Table_b1e0cba0d5f340a88739f9994705b2e7 [id=c67fc2b8004245ba93a726d85b0a8d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.525030 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70ba88bd1f84247b9e18659d7112cc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:31.524669 1989 log.cc:927] Time spent T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20250625 07:52:31.525449 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ada5281a2e4bf682c889778b478815. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.526597 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef8838908e649d995e0131de3130dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.526623 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d320734990404d1e9222bea1d88a18fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.527957 985 leader_election.cc:304] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.528344 2028 raft_consensus.cc:2747] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.529105 1161 ts_tablet_manager.cc:1918] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.529105 888 tablet_service.cc:1468] Processing CreateTablet for tablet 08521c443c284f6aac4342b8ceb24d86 (DEFAULT_TABLE table=default.Table_b649eef37b514286a98a4bc732c1870a [id=8e8a5851c650460b9162c7edf8682a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.529213 1161 log.cc:1199] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/651c74bf9a9b47d6ab5a875a5c304783
I20250625 07:52:31.529433 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08521c443c284f6aac4342b8ceb24d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.529520 1161 ts_tablet_manager.cc:1939] T 651c74bf9a9b47d6ab5a875a5c304783 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.526386 880 tablet_service.cc:1468] Processing CreateTablet for tablet 27e94c4fe10d48368a212eac8d8d09aa (DEFAULT_TABLE table=default.Table_b8146a0332fc4ef897cbf4a745c83b5f [id=846cd607927e405aadbaca92e012c507]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.533092 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e94c4fe10d48368a212eac8d8d09aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.533676 1166 tablet_replica.cc:331] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:31.533850 1166 raft_consensus.cc:2241] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.534001 1166 raft_consensus.cc:2270] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.534492 1166 ts_tablet_manager.cc:1905] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.534749 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7f196473315441c6bf8b5173ed46f833 (table default.Table_be2a902089674d54ad4be0f8d749236f [id=271cc904835e4b598070967bb42b59c6]) successfully deleted
I20250625 07:52:31.537240 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d661545e924905afa3d08eccac2e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.540971 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1017) took 1038 ms. Trace:
I20250625 07:52:31.541072 1161 rpcz_store.cc:276] 0625 07:52:30.502132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.502553 (+ 421us) service_pool.cc:225] Handling call
0625 07:52:31.511199 (+1008646us) ts_tablet_manager.cc:1145] Deleting tablet 651c74bf9a9b47d6ab5a875a5c304783
0625 07:52:31.511207 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.516975 (+ 5768us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.529010 (+ 12035us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.540950 (+ 11940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008521}
I20250625 07:52:31.541477 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1018) took 1008 ms. Trace:
I20250625 07:52:31.541563 1161 rpcz_store.cc:276] 0625 07:52:30.533149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.533208 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:31.541356 (+1008148us) ts_tablet_manager.cc:1145] Deleting tablet d0a75906385744e5ba6938f451a2d7ec
0625 07:52:31.541365 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.541458 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007963}
I20250625 07:52:31.541867 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1019) took 1002 ms. Trace:
I20250625 07:52:31.541939 1161 rpcz_store.cc:276] 0625 07:52:30.539261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.539322 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:31.541782 (+1002460us) ts_tablet_manager.cc:1145] Deleting tablet 9f4359fd842a4760bae75169e8914f1c
0625 07:52:31.541790 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.541854 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1002273}
I20250625 07:52:31.542413 1161 tablet_replica.cc:331] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.542477 915 raft_consensus.cc:2466] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:31.542604 1161 raft_consensus.cc:2241] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.542761 1161 raft_consensus.cc:2270] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.544297 1160 ts_tablet_manager.cc:1918] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.544780 1160 log.cc:1199] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9f4359fd842a4760bae75169e8914f1c
I20250625 07:52:31.545854 1160 ts_tablet_manager.cc:1939] T 9f4359fd842a4760bae75169e8914f1c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:31.546172 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1052) took 1006 ms. Trace:
I20250625 07:52:31.546242 1160 rpcz_store.cc:276] 0625 07:52:30.539990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.540048 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:31.149197 (+609149us) ts_tablet_manager.cc:1145] Deleting tablet 9f4359fd842a4760bae75169e8914f1c
0625 07:52:31.149209 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.402812 (+253603us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.544183 (+141371us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.546156 (+ 1973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":608950}
I20250625 07:52:31.547009 1160 tablet_replica.cc:331] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.547201 1160 raft_consensus.cc:2241] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.547354 1160 raft_consensus.cc:2270] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.547927 1160 ts_tablet_manager.cc:1905] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.550176 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 651c74bf9a9b47d6ab5a875a5c304783 (table default.Table_fa9514049a2142ca95833a0e0a0bf14b [id=2d0ea1f3d47e482c9f04d78c0db03a9e]) successfully deleted
W20250625 07:52:31.550554 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d0a75906385744e5ba6938f451a2d7ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0a75906385744e5ba6938f451a2d7ec already in progress: creating tablet
W20250625 07:52:31.550865 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9f4359fd842a4760bae75169e8914f1c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4359fd842a4760bae75169e8914f1c already in progress: creating tablet
W20250625 07:52:31.551167 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5986bb52335c46bf845c48cb044495b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5986bb52335c46bf845c48cb044495b1 already in progress: creating tablet
W20250625 07:52:31.551425 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7ef2eb85358b414c9755f81f97beaddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef2eb85358b414c9755f81f97beaddb already in progress: creating tablet
I20250625 07:52:31.552006 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5d251926c3c04d45b103b59b4c74db23 (DEFAULT_TABLE table=default.Table_8a81d60515d647b48496abdf8ef41532 [id=06374f1dde324a0ba96c7bb6c56de7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.552388 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d251926c3c04d45b103b59b4c74db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.553184 890 tablet_service.cc:1468] Processing CreateTablet for tablet 1748e9e9bdbc453da0c3ff120ae70ffd (DEFAULT_TABLE table=default.Table_9a56722f526b4aceb8716901c46a8be5 [id=73b185c4180549f194a489e39c652249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.553478 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1748e9e9bdbc453da0c3ff120ae70ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.554179 877 tablet_service.cc:1468] Processing CreateTablet for tablet 76ed33a26e764dba80b601edf5454600 (DEFAULT_TABLE table=default.Table_37ecb82090564313862912776a682691 [id=8306fec40b4648cb9aaf36094ff5d69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.554517 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76ed33a26e764dba80b601edf5454600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.556066 881 tablet_service.cc:1468] Processing CreateTablet for tablet 36c61f71a1674957805a48ccd6d0d4b3 (DEFAULT_TABLE table=default.Table_86b56ad960e1457ca74b3db973f6cbd0 [id=a7da47d868124bfbbf316a81083f438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.556407 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c61f71a1674957805a48ccd6d0d4b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.556746 884 tablet_service.cc:1468] Processing CreateTablet for tablet 363fee07295c44139fd713b33c82623c (DEFAULT_TABLE table=default.Table_2f21fd181ae64529a96b8dd7b83c8cad [id=c1e08f7897eb49a3a0d6c4b257e1a5c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.557066 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363fee07295c44139fd713b33c82623c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.557205 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 308b387d366c4d7884898937a9fc7690 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.556406 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.557505 883 tablet_service.cc:1468] Processing CreateTablet for tablet 54c2a9e362a6432880f30fd9e8719443 (DEFAULT_TABLE table=default.Table_b4fdd7ed1d8d4f2d814d8e22fea75b12 [id=c02ad9604e7240858e54579966bede5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.557796 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54c2a9e362a6432880f30fd9e8719443. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.557752 2028 raft_consensus.cc:491] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.557884 2028 raft_consensus.cc:513] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.552910 879 tablet_service.cc:1468] Processing CreateTablet for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 (DEFAULT_TABLE table=default.Table_e046fc4755bc40769ba73e88edfe6b53 [id=24c507fc082a44ef8e0e330294916b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.558300 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.558480 2028 leader_election.cc:290] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.559410 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c8abe36eab5498eb9bc3a0355101dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.559605 892 tablet_service.cc:1468] Processing CreateTablet for tablet e538d6dd213f4b66870d314dadbf5c9a (DEFAULT_TABLE table=default.Table_342146f029294bdaa3d7e68bc9e8d6b2 [id=33863857a134475a8f0b0d04a64c2836]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.559762 878 tablet_service.cc:1468] Processing CreateTablet for tablet b70ba88bd1f84247b9e18659d7112cc8 (DEFAULT_TABLE table=default.Table_30292e2cad9c4c039fce627374179f83 [id=6adf8fb3937741fbb867968ef8f6197f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.559952 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e538d6dd213f4b66870d314dadbf5c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.560215 889 tablet_service.cc:1468] Processing CreateTablet for tablet 82d661545e924905afa3d08eccac2e65 (DEFAULT_TABLE table=default.Table_54c26dc284d04c5b954ad6912d7345aa [id=b4839b762aa64498a595211b3f3d71e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.560540 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d661545e924905afa3d08eccac2e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.564884 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70ba88bd1f84247b9e18659d7112cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.565025 895 tablet_service.cc:1468] Processing CreateTablet for tablet 09120e19ea0f4ec7b94c89d173d3609f (DEFAULT_TABLE table=default.Table_bb75ba8ad3674db084f085ca2c573442 [id=f0bc9284b37442faac1f5f0e4faea800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.565325 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09120e19ea0f4ec7b94c89d173d3609f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.566699 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.567067 884 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef8838908e649d995e0131de3130dea (DEFAULT_TABLE table=default.Table_b333f5398ea94718a9f9c3208775b51d [id=8c823b8f9cba4a33af9c4c42bb0a0018]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.567369 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef8838908e649d995e0131de3130dea. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:31.569213 850 leader_election.cc:343] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.570958 850 leader_election.cc:304] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.571465 1793 raft_consensus.cc:2747] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.574023 894 tablet_service.cc:1468] Processing CreateTablet for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e (DEFAULT_TABLE table=default.Table_2e4dcd29c1264e7ead844712846cd17a [id=a4b8b5f15118470f8d0868221c941e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:31.576395 1166 ts_tablet_manager.cc:1918] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:31.581859 986 leader_election.cc:343] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 08bb47266e8b4a0fa5ec613560b8549e
I20250625 07:52:31.582700 1166 log.cc:1199] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:31.558902 1163 raft_consensus.cc:357] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:31.583796 985 leader_election.cc:343] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 08bb47266e8b4a0fa5ec613560b8549e
I20250625 07:52:31.583921 985 leader_election.cc:304] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:31.583997 850 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.584301 2028 raft_consensus.cc:2747] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.584981 1161 ts_tablet_manager.cc:1905] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.586997 1166 ts_tablet_manager.cc:1939] T 7e60d4f9aed44082b9905fbb44ffb110 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.586236 1158 tablet_bootstrap.cc:492] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.589737 1158 ts_tablet_manager.cc:1397] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.201s user 0.007s sys 0.005s
I20250625 07:52:31.590513 1158 raft_consensus.cc:357] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.590783 1158 raft_consensus.cc:383] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.590879 1158 raft_consensus.cc:738] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.591147 1158 consensus_queue.cc:260] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.591713 1158 ts_tablet_manager.cc:1428] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:31.592051 1158 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.592268 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 7be8ab5140aa4d48be978519cb99ed31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.593107 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:31.593827 1158 tablet_bootstrap.cc:654] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.583045 1163 raft_consensus.cc:383] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.604094 1163 raft_consensus.cc:738] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.604339 1163 consensus_queue.cc:260] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.609499 1161 ts_tablet_manager.cc:1918] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.611011 1161 log.cc:1199] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/802c22a6346442c88abbeb4b9ef20c57
W20250625 07:52:31.611020 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 49697a10cdac4c8ebb69f10f4c3437ec (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 49697a10cdac4c8ebb69f10f4c3437ec consensus state has no leader
I20250625 07:52:31.611557 1161 ts_tablet_manager.cc:1939] T 802c22a6346442c88abbeb4b9ef20c57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.612366 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1022) took 1049 ms. Trace:
I20250625 07:52:31.612865 1161 rpcz_store.cc:276] 0625 07:52:30.562902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.562962 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:31.542371 (+979409us) ts_tablet_manager.cc:1145] Deleting tablet 802c22a6346442c88abbeb4b9ef20c57
0625 07:52:31.542378 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.606495 (+ 64117us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.609419 (+ 2924us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.612347 (+ 2928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":979246}
I20250625 07:52:31.613583 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 802c22a6346442c88abbeb4b9ef20c57 (table default.Table_56e7b5805a334eb386aa31e60ece4da8 [id=ffc70c11b4ff4e75b849c6952d046979]) successfully deleted
I20250625 07:52:31.614066 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1023) took 1038 ms. Trace:
I20250625 07:52:31.614313 850 leader_election.cc:304] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.614737 1793 raft_consensus.cc:2747] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:31.615347 850 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.615514 850 leader_election.cc:304] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.615762 1793 raft_consensus.cc:2747] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.616063 1161 rpcz_store.cc:276] 0625 07:52:30.575386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.575441 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:31.613948 (+1038507us) ts_tablet_manager.cc:1145] Deleting tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a
0625 07:52:31.613958 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.614051 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038282}
W20250625 07:52:31.616852 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a already in progress: creating tablet
I20250625 07:52:31.617244 1161 tablet_replica.cc:331] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.617632 1161 raft_consensus.cc:2241] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.618036 1161 raft_consensus.cc:2270] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.618786 1161 ts_tablet_manager.cc:1905] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:31.616146 850 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.624285 850 leader_election.cc:304] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.624558 1793 raft_consensus.cc:2747] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:31.626279 713 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7480c07af7ce4a3889bd8d42d7419d57
I20250625 07:52:31.626410 713 leader_election.cc:304] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.626855 713 leader_election.cc:304] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.627099 2045 raft_consensus.cc:2747] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.609234 1163 ts_tablet_manager.cc:1428] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.224s user 0.001s sys 0.001s
I20250625 07:52:31.627287 2045 raft_consensus.cc:2802] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:31.627566 1163 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:31.628090 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9f4359fd842a4760bae75169e8914f1c (table default.Table_56a7bd0b475e451b8afa385ff9915ba5 [id=914805a4f6684943a2e76c948377f326]) successfully deleted
W20250625 07:52:31.628348 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5986bb52335c46bf845c48cb044495b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5986bb52335c46bf845c48cb044495b1 already in progress: creating tablet
W20250625 07:52:31.628614 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7ef2eb85358b414c9755f81f97beaddb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef2eb85358b414c9755f81f97beaddb already in progress: creating tablet
W20250625 07:52:31.631371 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90d62b84a6bd43f48d4c4b6cd24c6f4a already in progress: creating tablet
I20250625 07:52:31.631561 1163 tablet_bootstrap.cc:654] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.634771 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c415f4b5afe4cbfa6210b2cf48bca3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:31.637807 2045 raft_consensus.cc:695] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:31.638094 1166 tablet_replica.cc:331] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:31.638291 1166 raft_consensus.cc:2241] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.638448 1166 raft_consensus.cc:2270] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.638967 1166 ts_tablet_manager.cc:1905] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.644583 2045 consensus_queue.cc:237] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.645371 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:31.645624 915 raft_consensus.cc:2466] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:31.654372 1161 ts_tablet_manager.cc:1918] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.654503 1161 log.cc:1199] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b5f917cfd2384a72920847f8b89358b9
I20250625 07:52:31.654801 1161 ts_tablet_manager.cc:1939] T b5f917cfd2384a72920847f8b89358b9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.655112 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1024) took 1020 ms. Trace:
I20250625 07:52:31.655203 1161 rpcz_store.cc:276] 0625 07:52:30.634458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.634525 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:31.617198 (+982673us) ts_tablet_manager.cc:1145] Deleting tablet b5f917cfd2384a72920847f8b89358b9
0625 07:52:31.617207 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.634078 (+ 16871us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.654279 (+ 20201us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.655095 (+ 816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":982496}
I20250625 07:52:31.655694 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b5f917cfd2384a72920847f8b89358b9 (table default.Table_41d1f73c2d534e5496df588509888b0f [id=9099a03139514a1987603114ee41438a]) successfully deleted
I20250625 07:52:31.656059 1158 tablet_bootstrap.cc:492] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.656191 1158 ts_tablet_manager.cc:1397] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.010s
I20250625 07:52:31.657191 1158 raft_consensus.cc:357] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.657477 1158 raft_consensus.cc:383] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.657605 1158 raft_consensus.cc:738] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.657960 1158 consensus_queue.cc:260] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.658819 1158 ts_tablet_manager.cc:1428] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:31.659483 1158 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.662323 1161 tablet_replica.cc:331] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.662513 1161 raft_consensus.cc:2241] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.662726 1161 raft_consensus.cc:2270] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.663218 1161 ts_tablet_manager.cc:1905] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.675549 1160 ts_tablet_manager.cc:1918] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.675676 1160 log.cc:1199] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:31.676075 1160 ts_tablet_manager.cc:1939] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:31.676514 1160 tablet_replica.cc:331] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.676720 1160 raft_consensus.cc:2241] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.676882 1160 raft_consensus.cc:2270] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.677395 1160 ts_tablet_manager.cc:1905] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.681201 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (table default.Table_e57f684466224dd5a4a8ff79408e1a4d [id=3dcf0706033c48ceaa1fdd5763284bf0]) successfully deleted
I20250625 07:52:31.681754 1158 tablet_bootstrap.cc:654] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.682528 1793 raft_consensus.cc:491] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.682653 1793 raft_consensus.cc:513] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.684372 1793 leader_election.cc:290] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.688252 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "faf049714a7e4d7eb947c29f272bbb7c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.689625 850 leader_election.cc:343] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.690316 1163 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:31.690441 1163 ts_tablet_manager.cc:1397] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.003s
I20250625 07:52:31.691154 1163 raft_consensus.cc:357] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.691298 1163 raft_consensus.cc:383] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.691339 1163 raft_consensus.cc:738] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.691516 1163 consensus_queue.cc:260] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.691912 1163 ts_tablet_manager.cc:1428] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:31.694897 1163 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:31.699393 1166 ts_tablet_manager.cc:1918] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.699491 1166 log.cc:1199] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3b9491eb269948a29888a7301ea75d42
I20250625 07:52:31.697839 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "faf049714a7e4d7eb947c29f272bbb7c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:31.699787 1166 ts_tablet_manager.cc:1939] T 3b9491eb269948a29888a7301ea75d42 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.699836 1162 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.699963 1162 ts_tablet_manager.cc:1397] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.280s user 0.007s sys 0.000s
W20250625 07:52:31.700965 851 leader_election.cc:343] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.701084 851 leader_election.cc:304] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.701566 1166 tablet_replica.cc:331] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:31.701745 1166 raft_consensus.cc:2241] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.701730 1793 raft_consensus.cc:2747] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.701865 1166 raft_consensus.cc:2270] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.702307 1166 ts_tablet_manager.cc:1905] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.702755 713 leader_election.cc:304] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.703003 2097 raft_consensus.cc:2802] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:31.703068 2097 raft_consensus.cc:491] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:31.703122 2097 raft_consensus.cc:3058] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:31.703791 1158 tablet_bootstrap.cc:492] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.703907 1158 ts_tablet_manager.cc:1397] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20250625 07:52:31.704064 1162 raft_consensus.cc:357] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.704212 1162 raft_consensus.cc:383] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.704258 1162 raft_consensus.cc:738] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.704434 1162 consensus_queue.cc:260] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.704526 1158 raft_consensus.cc:357] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.704706 1158 raft_consensus.cc:383] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.704764 1158 raft_consensus.cc:738] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.704942 1158 consensus_queue.cc:260] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.705607 1158 ts_tablet_manager.cc:1428] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:31.706131 1158 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.706207 2097 raft_consensus.cc:513] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.707266 2097 leader_election.cc:290] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.714046 1161 ts_tablet_manager.cc:1918] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.714490 1161 log.cc:1199] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/ba24bbd0c4694930978a7750f07601d9
I20250625 07:52:31.715546 1161 ts_tablet_manager.cc:1939] T ba24bbd0c4694930978a7750f07601d9 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.716339 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1025) took 1062 ms. Trace:
I20250625 07:52:31.719803 1158 tablet_bootstrap.cc:654] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.720281 1166 ts_tablet_manager.cc:1918] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.720394 1166 log.cc:1199] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/f191436f3b9244eaa12d211b6f125175
I20250625 07:52:31.720677 1166 ts_tablet_manager.cc:1939] T f191436f3b9244eaa12d211b6f125175 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.721123 1166 tablet_replica.cc:331] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:31.721302 1166 raft_consensus.cc:2241] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.721448 1166 raft_consensus.cc:2270] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.722088 1166 ts_tablet_manager.cc:1905] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.725739 1162 ts_tablet_manager.cc:1428] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20250625 07:52:31.726027 1162 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:31.726375 584 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:43317, user_credentials={real_user=slave}} blocked reactor thread for 322377us
I20250625 07:52:31.730154 1162 tablet_bootstrap.cc:654] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.732275 1166 ts_tablet_manager.cc:1918] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.732407 1166 log.cc:1199] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7480c07af7ce4a3889bd8d42d7419d57
I20250625 07:52:31.732792 1166 ts_tablet_manager.cc:1939] T 7480c07af7ce4a3889bd8d42d7419d57 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:31.735069 1162 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.735190 1162 ts_tablet_manager.cc:1397] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250625 07:52:31.735845 1162 raft_consensus.cc:357] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.736027 1162 raft_consensus.cc:383] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.736076 1162 raft_consensus.cc:738] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.736270 1162 consensus_queue.cc:260] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.736727 1162 ts_tablet_manager.cc:1428] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250625 07:52:31.736994 1162 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.738492 1162 tablet_bootstrap.cc:654] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.745177 1162 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.745307 1162 ts_tablet_manager.cc:1397] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250625 07:52:31.745995 1162 raft_consensus.cc:357] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.746156 1162 raft_consensus.cc:383] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.746201 1162 raft_consensus.cc:738] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.746435 1162 consensus_queue.cc:260] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.747085 1162 ts_tablet_manager.cc:1428] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:31.747318 1162 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.748963 1162 tablet_bootstrap.cc:654] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:31.752023 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 48711.1us
I20250625 07:52:31.753090 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (table default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]) successfully deleted
I20250625 07:52:31.754312 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:31.756152 713 leader_election.cc:343] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.757165 1047 raft_consensus.cc:3058] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:31.758133 1162 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.758253 1162 ts_tablet_manager.cc:1397] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20250625 07:52:31.758929 1162 raft_consensus.cc:357] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.759073 1162 raft_consensus.cc:383] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.759119 1162 raft_consensus.cc:738] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.759301 1162 consensus_queue.cc:260] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.759727 1162 ts_tablet_manager.cc:1428] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:31.759985 1162 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.716470 1161 rpcz_store.cc:276] 0625 07:52:30.654089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.654146 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:31.662276 (+1008130us) ts_tablet_manager.cc:1145] Deleting tablet ba24bbd0c4694930978a7750f07601d9
0625 07:52:31.662288 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.666299 (+ 4011us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.713598 (+ 47299us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.716319 (+ 2721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007950}
I20250625 07:52:31.760881 1161 tablet_replica.cc:331] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.761054 1161 raft_consensus.cc:2241] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.761072 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet ba24bbd0c4694930978a7750f07601d9 (table default.Table_ce3de32cf03042d6990e81e54f461669 [id=30de507c3fb344ecb071e9def7dcf119]) successfully deleted
I20250625 07:52:31.761188 1161 raft_consensus.cc:2270] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.761669 1162 tablet_bootstrap.cc:654] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.761703 1161 ts_tablet_manager.cc:1905] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.766521 1162 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.766654 1162 ts_tablet_manager.cc:1397] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250625 07:52:31.767417 1162 raft_consensus.cc:357] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.767575 1162 raft_consensus.cc:383] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.767659 1162 raft_consensus.cc:738] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.767849 1162 consensus_queue.cc:260] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.768290 1162 ts_tablet_manager.cc:1428] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:31.769246 1158 tablet_bootstrap.cc:492] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.769382 1158 ts_tablet_manager.cc:1397] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.003s
I20250625 07:52:31.770067 1158 raft_consensus.cc:357] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.770229 1158 raft_consensus.cc:383] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.770285 1158 raft_consensus.cc:738] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.770500 1158 consensus_queue.cc:260] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.770913 1158 ts_tablet_manager.cc:1428] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:31.771152 1158 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.785135 1162 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.788477 1162 tablet_bootstrap.cc:654] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.797755 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:31.799597 1163 tablet_bootstrap.cc:654] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.801141 1160 ts_tablet_manager.cc:1918] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.801254 1160 log.cc:1199] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/bbe5c37fc9bd49c685de8f469a7af0c6
I20250625 07:52:31.801538 1160 ts_tablet_manager.cc:1939] T bbe5c37fc9bd49c685de8f469a7af0c6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:31.801954 1160 tablet_replica.cc:331] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.802143 1160 raft_consensus.cc:2241] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.802305 1160 raft_consensus.cc:2270] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.802821 1160 ts_tablet_manager.cc:1905] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.806083 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5fe3ea5745743019d23d49ff808070a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:31.806927 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet bbe5c37fc9bd49c685de8f469a7af0c6 (table default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]) successfully deleted
I20250625 07:52:31.811713 1163 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:31.811875 1163 ts_tablet_manager.cc:1397] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.002s
I20250625 07:52:31.812507 1163 raft_consensus.cc:357] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.812646 1163 raft_consensus.cc:383] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.812708 1163 raft_consensus.cc:738] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.813043 1163 consensus_queue.cc:260] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.814179 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7e60d4f9aed44082b9905fbb44ffb110 (table default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]) successfully deleted
W20250625 07:52:31.814435 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 683954dc7ecc4185847f46a05fd2c53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683954dc7ecc4185847f46a05fd2c53d already in progress: creating tablet
W20250625 07:52:31.814606 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 652e07d46af74945bf504f7d9015a446 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652e07d46af74945bf504f7d9015a446 already in progress: creating tablet
W20250625 07:52:31.814790 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 78a1bfd87f084d50bd2b602b89c18a2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a1bfd87f084d50bd2b602b89c18a2f already in progress: creating tablet
I20250625 07:52:31.814947 1163 ts_tablet_manager.cc:1428] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250625 07:52:31.814966 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b782414a682246feb2811b4e3bf8b5a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b782414a682246feb2811b4e3bf8b5a2 already in progress: creating tablet
W20250625 07:52:31.815117 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bedc38018eb6412b9e6ca3f0ee34cf87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bedc38018eb6412b9e6ca3f0ee34cf87 already in progress: creating tablet
W20250625 07:52:31.815270 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 6881336f1ff44e3cb7d1bc8937e503a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6881336f1ff44e3cb7d1bc8937e503a4 already in progress: creating tablet
I20250625 07:52:31.815303 1163 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:31.815677 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3b9491eb269948a29888a7301ea75d42 (table default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]) successfully deleted
W20250625 07:52:31.815884 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0cc3ba09d5e4c3d88e74fffc97eea8b already in progress: creating tablet
I20250625 07:52:31.816148 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet f191436f3b9244eaa12d211b6f125175 (table default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]) successfully deleted
I20250625 07:52:31.816291 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7480c07af7ce4a3889bd8d42d7419d57 (table default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]) successfully deleted
W20250625 07:52:31.816457 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 08cd482ceec244c2bed2b116d5c14d89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cd482ceec244c2bed2b116d5c14d89 already in progress: creating tablet
W20250625 07:52:31.816632 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7c8abe36eab5498eb9bc3a0355101dd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c8abe36eab5498eb9bc3a0355101dd6 already in progress: creating tablet
I20250625 07:52:31.817021 1163 tablet_bootstrap.cc:654] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.772678 1158 tablet_bootstrap.cc:654] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.822449 915 raft_consensus.cc:3058] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:31.823424 1162 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.823575 1162 ts_tablet_manager.cc:1397] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250625 07:52:31.824486 1162 raft_consensus.cc:357] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.824719 1162 raft_consensus.cc:383] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.824777 1162 raft_consensus.cc:738] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.824982 1162 consensus_queue.cc:260] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.825219 915 raft_consensus.cc:2466] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:31.825431 1162 ts_tablet_manager.cc:1428] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:31.826340 713 leader_election.cc:304] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.826581 2045 raft_consensus.cc:2802] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:31.826766 2045 raft_consensus.cc:695] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:31.826988 2045 consensus_queue.cc:237] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.831029 1162 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.832641 1162 tablet_bootstrap.cc:654] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:31.837826 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 308b387d366c4d7884898937a9fc7690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308b387d366c4d7884898937a9fc7690 already in progress: creating tablet
W20250625 07:52:31.838166 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 484a49fab6134d059c64f5a3f2b29e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484a49fab6134d059c64f5a3f2b29e03 already in progress: creating tablet
W20250625 07:52:31.838521 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7be8ab5140aa4d48be978519cb99ed31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be8ab5140aa4d48be978519cb99ed31 already in progress: creating tablet
I20250625 07:52:31.840407 1162 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.840543 1162 ts_tablet_manager.cc:1397] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250625 07:52:31.841302 1162 raft_consensus.cc:357] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.841454 1162 raft_consensus.cc:383] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.841498 1162 raft_consensus.cc:738] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.841683 1162 consensus_queue.cc:260] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.842089 1162 ts_tablet_manager.cc:1428] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250625 07:52:31.842314 1162 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.844812 1162 tablet_bootstrap.cc:654] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:31.845422 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 168f1a13533045d89ba1f07a72d4ff63 consensus state has no leader
W20250625 07:52:31.846063 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2082e5e28c5c4d1fac7e40acf77f75e6 consensus state has no leader
I20250625 07:52:31.849419 2131 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:31.856791 1160 ts_tablet_manager.cc:1918] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.856927 1160 log.cc:1199] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/547b758c2e6c4558bb232f14b37aa81a
I20250625 07:52:31.857234 1160 ts_tablet_manager.cc:1939] T 547b758c2e6c4558bb232f14b37aa81a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:31.859601 1158 tablet_bootstrap.cc:492] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.859797 1158 ts_tablet_manager.cc:1397] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.004s
I20250625 07:52:31.860503 1158 raft_consensus.cc:357] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.860680 1158 raft_consensus.cc:383] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.860770 1158 raft_consensus.cc:738] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.860963 1158 consensus_queue.cc:260] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.861413 1158 ts_tablet_manager.cc:1428] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:31.865303 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1058) took 1016 ms. Trace:
I20250625 07:52:31.865422 1160 rpcz_store.cc:276] 0625 07:52:30.849160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.849361 (+ 201us) service_pool.cc:225] Handling call
0625 07:52:31.801912 (+952551us) ts_tablet_manager.cc:1145] Deleting tablet 547b758c2e6c4558bb232f14b37aa81a
0625 07:52:31.801920 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.837565 (+ 35645us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.856631 (+ 19066us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.865272 (+ 8641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":952383}
I20250625 07:52:31.865944 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 547b758c2e6c4558bb232f14b37aa81a (table default.Table_0c0f428f2b54479a8e0bd1f36ef2dede [id=83522331e5df4bafaa65c81be08c7525]) successfully deleted
I20250625 07:52:31.866782 1160 tablet_replica.cc:331] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:31.868579 1160 raft_consensus.cc:2241] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.869124 1160 raft_consensus.cc:2270] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.871068 1161 ts_tablet_manager.cc:1918] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:31.871237 1161 log.cc:1199] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a7cb2bcdd07c42e680eb4f155212423f
I20250625 07:52:31.871759 1161 ts_tablet_manager.cc:1939] T a7cb2bcdd07c42e680eb4f155212423f P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:31.874375 1160 ts_tablet_manager.cc:1905] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:31.880251 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f406d1c8777f474c97f6e804cb230bb6 consensus state has no leader
I20250625 07:52:31.886673 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 64ca83369ffe46ea9bd756f20da936cf
I20250625 07:52:31.887285 593 catalog_manager.cc:6652] ReplaceTablet: tablet 64ca83369ffe46ea9bd756f20da936cf deleted and replaced by tablet b786ef91745c47638af90553a66fffdf
I20250625 07:52:31.889540 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 64ca83369ffe46ea9bd756f20da936cf with delete_type TABLET_DATA_DELETED (replaced by tablet b786ef91745c47638af90553a66fffdf) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.890293 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 64ca83369ffe46ea9bd756f20da936cf with delete_type TABLET_DATA_DELETED (replaced by tablet b786ef91745c47638af90553a66fffdf) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:31.891498 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_381f2b07c1fe4a47adbf6b5f45f738b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.892318 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_381f2b07c1fe4a47adbf6b5f45f738b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:31.892729 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 64ca83369ffe46ea9bd756f20da936cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64ca83369ffe46ea9bd756f20da936cf already in progress: creating tablet
I20250625 07:52:31.895781 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_00236ecf6f4146f2be402a5a04cb0d10" } new_table_name: "default.Table_d2b9caad298b479eb23cd9548220772d" modify_external_catalogs: true
I20250625 07:52:31.897563 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3c26e45d95314578aa155e978ffd1096" } new_table_name: "default.Table_a58fb2c3c7e84ac9803c717ac5e9d8e1" modify_external_catalogs: true
I20250625 07:52:31.899031 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cf47515a30a347929633a53b20a19b4b" } modify_external_catalogs: true
I20250625 07:52:31.899245 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cf47515a30a347929633a53b20a19b4b" } modify_external_catalogs: true
I20250625 07:52:31.903600 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1027) took 1073 ms. Trace:
I20250625 07:52:31.903735 1161 rpcz_store.cc:276] 0625 07:52:30.830551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.830599 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:31.760842 (+930243us) ts_tablet_manager.cc:1145] Deleting tablet a7cb2bcdd07c42e680eb4f155212423f
0625 07:52:31.760850 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.811077 (+ 50227us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.870962 (+ 59885us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:31.903560 (+ 32598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":930062}
I20250625 07:52:31.905063 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3a28b5bd95a545b2bde25fce57861fc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:31.905267 2028 raft_consensus.cc:491] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:31.905530 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a28b5bd95a545b2bde25fce57861fc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.905428 2028 raft_consensus.cc:513] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.906230 2028 leader_election.cc:290] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:31.907449 986 leader_election.cc:343] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:31.908789 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1029) took 1062 ms. Trace:
I20250625 07:52:31.909050 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3d0f6d3d892d4afdbb88b6ed05592235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.909514 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d0f6d3d892d4afdbb88b6ed05592235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.910075 1158 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.909402 1161 rpcz_store.cc:276] 0625 07:52:30.846193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.846239 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:31.904238 (+1057999us) ts_tablet_manager.cc:1145] Deleting tablet 652e07d46af74945bf504f7d9015a446
0625 07:52:31.904271 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.908514 (+ 4243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024919}
I20250625 07:52:31.912858 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1028) took 1066 ms. Trace:
I20250625 07:52:31.913306 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:31.914330 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_000e46983c3d4c018a661c9efae36202" } modify_external_catalogs: true
I20250625 07:52:31.914470 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_000e46983c3d4c018a661c9efae36202" } modify_external_catalogs: true
I20250625 07:52:31.913590 1161 rpcz_store.cc:276] 0625 07:52:30.846056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.846861 (+ 805us) service_pool.cc:225] Handling call
0625 07:52:31.911688 (+1064827us) ts_tablet_manager.cc:1145] Deleting tablet 683954dc7ecc4185847f46a05fd2c53d
0625 07:52:31.911927 (+ 239us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.912541 (+ 614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1032299}
I20250625 07:52:31.916486 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f3cd40e7670e492bba639148deda5ffd" } modify_external_catalogs: true
I20250625 07:52:31.916733 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f3cd40e7670e492bba639148deda5ffd" } modify_external_catalogs: true
I20250625 07:52:31.917304 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_72e90a4803a04ea09a7b37b41bc7b65a" } modify_external_catalogs: true
I20250625 07:52:31.917443 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_72e90a4803a04ea09a7b37b41bc7b65a" } modify_external_catalogs: true
I20250625 07:52:31.917640 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1030) took 1069 ms. Trace:
I20250625 07:52:31.918323 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a7cb2bcdd07c42e680eb4f155212423f (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
W20250625 07:52:31.918545 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 652e07d46af74945bf504f7d9015a446 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652e07d46af74945bf504f7d9015a446 already in progress: creating tablet
W20250625 07:52:31.918758 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 683954dc7ecc4185847f46a05fd2c53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683954dc7ecc4185847f46a05fd2c53d already in progress: creating tablet
I20250625 07:52:31.919205 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 08bb47266e8b4a0fa5ec613560b8549e with delete_type TABLET_DATA_DELETED (replaced by tablet a25f89bffcd0430a976113d512c6a699) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.918607 1161 rpcz_store.cc:276] 0625 07:52:30.848420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.848827 (+ 407us) service_pool.cc:225] Handling call
0625 07:52:31.916586 (+1067759us) ts_tablet_manager.cc:1145] Deleting tablet 78a1bfd87f084d50bd2b602b89c18a2f
0625 07:52:31.916598 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.917615 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036860}
W20250625 07:52:31.920809 985 leader_election.cc:343] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:31.920920 985 leader_election.cc:304] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.921136 2028 raft_consensus.cc:2747] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:31.921300 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 78a1bfd87f084d50bd2b602b89c18a2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a1bfd87f084d50bd2b602b89c18a2f already in progress: creating tablet
I20250625 07:52:31.922281 2028 raft_consensus.cc:491] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.922438 2028 raft_consensus.cc:513] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.923243 2028 leader_election.cc:290] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.924175 1158 tablet_bootstrap.cc:654] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:31.924364 986 leader_election.cc:343] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 31942f0998bb4118a91f476325851d5c
I20250625 07:52:31.925902 1163 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:31.926074 1163 ts_tablet_manager.cc:1397] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.003s
W20250625 07:52:31.926604 985 leader_election.cc:343] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 31942f0998bb4118a91f476325851d5c
I20250625 07:52:31.926713 985 leader_election.cc:304] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.926956 2028 raft_consensus.cc:2747] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.930115 1163 raft_consensus.cc:357] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.930315 1163 raft_consensus.cc:383] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.930385 1163 raft_consensus.cc:738] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.923457 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1031) took 1071 ms. Trace:
I20250625 07:52:31.930603 1161 rpcz_store.cc:276] 0625 07:52:30.851163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.851691 (+ 528us) service_pool.cc:225] Handling call
0625 07:52:31.921984 (+1070293us) ts_tablet_manager.cc:1145] Deleting tablet b782414a682246feb2811b4e3bf8b5a2
0625 07:52:31.922287 (+ 303us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.923038 (+ 751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042447}
I20250625 07:52:31.930676 1163 consensus_queue.cc:260] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.930972 2112 raft_consensus.cc:491] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.931118 2112 raft_consensus.cc:513] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.931751 2112 leader_election.cc:290] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:31.932399 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:31.933049 851 leader_election.cc:343] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.932968 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1034) took 1081 ms. Trace:
I20250625 07:52:31.933406 1161 rpcz_store.cc:276] 0625 07:52:30.851469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.852158 (+ 689us) service_pool.cc:225] Handling call
0625 07:52:31.932280 (+1080122us) ts_tablet_manager.cc:1145] Deleting tablet bedc38018eb6412b9e6ca3f0ee34cf87
0625 07:52:31.932398 (+ 118us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.932829 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052358}
I20250625 07:52:31.934403 1163 ts_tablet_manager.cc:1428] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:31.934369 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.934791 850 leader_election.cc:343] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.934981 850 leader_election.cc:304] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.936097 2112 raft_consensus.cc:2747] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.940802 595 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_99eaf4f7bbec455ab0c9291fedfa221d [id=23322aa9321140c39272b5b2a102fb02] alter complete (version 4)
W20250625 07:52:31.944577 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e97e1232b70d4a17829aa199f35268e8 has no consensus state
I20250625 07:52:31.934681 1163 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:31.931351 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b782414a682246feb2811b4e3bf8b5a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b782414a682246feb2811b4e3bf8b5a2 already in progress: creating tablet
W20250625 07:52:31.946151 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bedc38018eb6412b9e6ca3f0ee34cf87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bedc38018eb6412b9e6ca3f0ee34cf87 already in progress: creating tablet
I20250625 07:52:31.936154 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1040) took 1076 ms. Trace:
I20250625 07:52:31.946368 2028 raft_consensus.cc:491] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.946623 2028 raft_consensus.cc:513] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:31.946847 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75e882e974da4e929c1301b56fbf66fa consensus state has no leader
I20250625 07:52:31.947537 2028 leader_election.cc:290] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:31.948396 985 leader_election.cc:343] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a0518fbc7f52454296ae79a3c92ef800
W20250625 07:52:31.948905 986 leader_election.cc:343] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a0518fbc7f52454296ae79a3c92ef800
I20250625 07:52:31.949072 986 leader_election.cc:304] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:31.946544 1161 rpcz_store.cc:276] 0625 07:52:30.858957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.859121 (+ 164us) service_pool.cc:225] Handling call
0625 07:52:31.935055 (+1075934us) ts_tablet_manager.cc:1145] Deleting tablet 6881336f1ff44e3cb7d1bc8937e503a4
0625 07:52:31.935066 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:31.935682 (+ 616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054297}
I20250625 07:52:31.949553 2028 raft_consensus.cc:2747] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:31.949791 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6881336f1ff44e3cb7d1bc8937e503a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6881336f1ff44e3cb7d1bc8937e503a4 already in progress: creating tablet
W20250625 07:52:31.950533 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0cc3ba09d5e4c3d88e74fffc97eea8b already in progress: creating tablet
I20250625 07:52:31.952337 1163 tablet_bootstrap.cc:654] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.953879 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet d4c744fef0e24cb39c4bb0787adb1cbc (table default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071]). Expected version 1 got 0
I20250625 07:52:31.954484 1162 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:31.954649 1162 ts_tablet_manager.cc:1397] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.112s user 0.025s sys 0.071s
I20250625 07:52:31.955384 1162 raft_consensus.cc:357] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.955554 1162 raft_consensus.cc:383] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.955617 1162 raft_consensus.cc:738] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.955816 1162 consensus_queue.cc:260] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.955986 590 master_service.cc:948] ReplaceTablet: received request to replace tablet 4f1c150fb54c405c8dc61ac133d98b13 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:31.956238 1162 ts_tablet_manager.cc:1428] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:31.956503 1162 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:31.957015 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a0518fbc7f52454296ae79a3c92ef800 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.957271 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 445bfed532004108a12c8fd78370c1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.957497 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a5a3751333bc48fc8a59b0e7b67a9935 with delete_type TABLET_DATA_DELETED (replaced by tablet f9a94dc1432d4cbdbf3eecacb1429c23) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.957751 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9418b81f010d4f369d3409e9364bc5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.958000 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 960ded22e2eb42d9a3494c020a4ea354 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.958097 1161 tablet_replica.cc:331] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:31.958349 1161 raft_consensus.cc:2241] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:31.958564 1161 raft_consensus.cc:2270] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:31.959510 1162 tablet_bootstrap.cc:654] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:31.960417 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 7e60d4f9aed44082b9905fbb44ffb110 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:31.960958 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dc62a435b6d44031bc3e5a60a3914e8f" } modify_external_catalogs: true
I20250625 07:52:31.961133 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dc62a435b6d44031bc3e5a60a3914e8f" } modify_external_catalogs: true
I20250625 07:52:31.961565 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.962057 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_56030f59057b4ce39f60fb888dd0fc17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:31.962646 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56030f59057b4ce39f60fb888dd0fc17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.962973 1161 ts_tablet_manager.cc:1905] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:31.964083 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a9b7236721334c269477fa34a36330b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:31.965960 1158 tablet_bootstrap.cc:492] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:31.966145 1158 ts_tablet_manager.cc:1397] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.002s
I20250625 07:52:31.967181 1158 raft_consensus.cc:357] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.967446 1158 raft_consensus.cc:383] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:31.967545 1158 raft_consensus.cc:738] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:31.967926 1158 consensus_queue.cc:260] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:31.969341 876 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:31.969607 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9b7236721334c269477fa34a36330b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:31.968958 1158 ts_tablet_manager.cc:1428] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:31.972044 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.973147 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.974493 1158 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:31.978603 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:31.979121 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 250175f54ba94a9fbc2455d9cf27fb2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:31.979584 1850 raft_consensus.cc:491] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.979751 1850 raft_consensus.cc:513] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.980566 1850 leader_election.cc:290] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.981505 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccddd630a82341cba7a857e67a349fe0" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:31.982048 713 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:31.982753 713 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:31.982870 713 leader_election.cc:304] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.983098 2045 raft_consensus.cc:2747] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:31.988559 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 64ca83369ffe46ea9bd756f20da936cf with delete_type TABLET_DATA_DELETED (replaced by tablet b786ef91745c47638af90553a66fffdf) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:31.992722 2045 raft_consensus.cc:491] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:31.992966 2045 raft_consensus.cc:513] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:31.994140 2045 leader_election.cc:290] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:31.994948 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:31.994961 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:31.995148 914 raft_consensus.cc:2466] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:31.995553 713 leader_election.cc:343] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:31.996434 713 leader_election.cc:304] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:31.996738 2045 raft_consensus.cc:2802] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:31.996865 2045 raft_consensus.cc:491] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:31.996954 2045 raft_consensus.cc:3058] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.001329 2045 raft_consensus.cc:513] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.002221 2045 leader_election.cc:290] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.005739 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:32.006366 713 leader_election.cc:343] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.009714 1160 ts_tablet_manager.cc:1918] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.009842 1160 log.cc:1199] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:32.010131 1160 ts_tablet_manager.cc:1939] T d393a85f5fde4b3fb57e5b3ec372791c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:32.010905 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1059) took 1154 ms. Trace:
I20250625 07:52:32.011001 1160 rpcz_store.cc:276] 0625 07:52:30.856865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.856918 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:31.866729 (+1009811us) ts_tablet_manager.cc:1145] Deleting tablet d393a85f5fde4b3fb57e5b3ec372791c
0625 07:52:31.866740 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.000246 (+133506us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.009577 (+ 9331us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.010881 (+ 1304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009632}
I20250625 07:52:32.011341 1160 tablet_replica.cc:331] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:32.011525 1160 raft_consensus.cc:2241] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.011864 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d393a85f5fde4b3fb57e5b3ec372791c (table default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]) successfully deleted
I20250625 07:52:32.011705 1160 raft_consensus.cc:2270] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.016866 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:32.017174 914 raft_consensus.cc:3058] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.016901 1160 ts_tablet_manager.cc:1905] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.017551 1163 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.017719 1163 ts_tablet_manager.cc:1397] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20250625 07:52:32.018523 1163 raft_consensus.cc:357] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.018710 1163 raft_consensus.cc:383] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.018779 1163 raft_consensus.cc:738] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.018991 1163 consensus_queue.cc:260] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.019447 1163 ts_tablet_manager.cc:1428] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:32.020061 1163 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.020576 914 raft_consensus.cc:2466] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:32.021167 713 leader_election.cc:304] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.021453 2045 raft_consensus.cc:2802] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:32.021579 2045 raft_consensus.cc:695] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:32.021749 1163 tablet_bootstrap.cc:654] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.021853 2045 consensus_queue.cc:237] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.027683 1158 tablet_bootstrap.cc:654] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.033087 1163 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.033241 1163 ts_tablet_manager.cc:1397] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20250625 07:52:32.033998 1163 raft_consensus.cc:357] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.034184 1163 raft_consensus.cc:383] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.034250 1163 raft_consensus.cc:738] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.034478 1163 consensus_queue.cc:260] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.034790 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.035094 1163 ts_tablet_manager.cc:1428] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:32.035434 1163 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.037410 1163 tablet_bootstrap.cc:654] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.043102 1162 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.043277 1162 ts_tablet_manager.cc:1397] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.087s user 0.027s sys 0.035s
I20250625 07:52:32.044265 1162 raft_consensus.cc:357] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.044476 1162 raft_consensus.cc:383] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.044549 1162 raft_consensus.cc:738] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.044806 1162 consensus_queue.cc:260] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.045393 1162 ts_tablet_manager.cc:1428] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:32.045673 1162 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.048805 1162 tablet_bootstrap.cc:654] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.050199 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6208679543004885b54f1f20d7f98f9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.050773 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6208679543004885b54f1f20d7f98f9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.051961 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5c1c689b7e634891a7e82c704c095684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.052253 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
W20250625 07:52:32.052445 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c1c689b7e634891a7e82c704c095684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.053539 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f7ce218193d24d2b96f16895ea6ac04a" } new_table_name: "default.Table_44ec701e9933466cac766250eb9d230e" modify_external_catalogs: true
I20250625 07:52:32.057310 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 51c584e6bffb4f0a8020dadf78d718a7
I20250625 07:52:32.058729 597 catalog_manager.cc:5582] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:32.059060 1162 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.059091 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
I20250625 07:52:32.059198 1162 ts_tablet_manager.cc:1397] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20250625 07:52:32.059326 597 catalog_manager.cc:5582] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:32.059669 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:32.059824 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]). Expected version 1 got 0
I20250625 07:52:32.059993 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:32.059928 1162 raft_consensus.cc:357] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.060106 1162 raft_consensus.cc:383] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.060163 1162 raft_consensus.cc:738] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.060360 1162 consensus_queue.cc:260] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.060603 1850 raft_consensus.cc:491] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.060745 1850 raft_consensus.cc:513] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.061239 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5099f01a4a124d23a7c32accbd576e54
I20250625 07:52:32.061501 1850 leader_election.cc:290] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.062001 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1baf94a051e485eaf4c0e074e147252" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.062578 713 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.062641 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:32.062384 1162 ts_tablet_manager.cc:1428] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:32.063161 1162 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:32.063570 713 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:32.063680 713 leader_election.cc:304] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.063974 1850 raft_consensus.cc:2747] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.064713 749 tablet_service.cc:1468] Processing CreateTablet for tablet 24c767b82ed24df1b145ff2df33ba155 (DEFAULT_TABLE table=default.Table_57bbb43218994668b3c285c21c088ffa [id=0b372231411b4a27bd7788e4fe589885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.065133 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c767b82ed24df1b145ff2df33ba155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.064684 756 tablet_service.cc:1468] Processing CreateTablet for tablet 46f81617a460458791e215b81c589241 (DEFAULT_TABLE table=default.Table_143a766c3eb149338d9b90575b6deaf1 [id=044d275c5fbb468db72885400493f0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.065426 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f81617a460458791e215b81c589241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.065663 752 tablet_service.cc:1468] Processing CreateTablet for tablet 1842b67d00e248c59e02c1603a8c5300 (DEFAULT_TABLE table=default.Table_04ba3f20d1184e0c9f9deb21ea849289 [id=2ff5265510874cacaf03166ff69e2469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.065974 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1842b67d00e248c59e02c1603a8c5300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.065945 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 24c767b82ed24df1b145ff2df33ba155 (DEFAULT_TABLE table=default.Table_57bbb43218994668b3c285c21c088ffa [id=0b372231411b4a27bd7788e4fe589885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.066298 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c767b82ed24df1b145ff2df33ba155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.066278 754 tablet_service.cc:1468] Processing CreateTablet for tablet fd3ea59a079f4e05a2ee540dde05562b (DEFAULT_TABLE table=default.Table_0b3fc22526374bb1a70819a649be9dfe [id=c34ff2769412405d976aee0cfad52f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.066577 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3ea59a079f4e05a2ee540dde05562b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.068800 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 5099f01a4a124d23a7c32accbd576e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.068820 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.064790 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.068948 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 46f81617a460458791e215b81c589241 (DEFAULT_TABLE table=default.Table_143a766c3eb149338d9b90575b6deaf1 [id=044d275c5fbb468db72885400493f0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.069268 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f81617a460458791e215b81c589241. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:32.070089 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5099f01a4a124d23a7c32accbd576e54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5099f01a4a124d23a7c32accbd576e54 already in progress: creating tablet
W20250625 07:52:32.070329 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 51c584e6bffb4f0a8020dadf78d718a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c584e6bffb4f0a8020dadf78d718a7 already in progress: creating tablet
I20250625 07:52:32.072837 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.072973 1014 tablet_service.cc:1468] Processing CreateTablet for tablet fd3ea59a079f4e05a2ee540dde05562b (DEFAULT_TABLE table=default.Table_0b3fc22526374bb1a70819a649be9dfe [id=c34ff2769412405d976aee0cfad52f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.073325 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3ea59a079f4e05a2ee540dde05562b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.074630 2112 raft_consensus.cc:491] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:32.074730 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3cdbdd1e3b734463ab452bafecb75822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdbdd1e3b734463ab452bafecb75822 already in progress: creating tablet
I20250625 07:52:32.074770 2112 raft_consensus.cc:513] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.075410 2112 leader_election.cc:290] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.076000 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.076495 851 leader_election.cc:343] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.076958 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1842b67d00e248c59e02c1603a8c5300 (DEFAULT_TABLE table=default.Table_04ba3f20d1184e0c9f9deb21ea849289 [id=2ff5265510874cacaf03166ff69e2469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.077306 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1842b67d00e248c59e02c1603a8c5300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.077745 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5099f01a4a124d23a7c32accbd576e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.077941 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.078199 2028 raft_consensus.cc:491] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.078310 2028 raft_consensus.cc:513] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.078420 1161 ts_tablet_manager.cc:1918] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.078569 1161 log.cc:1199] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/10c7f4b4187c428bbec136c85219da4d
I20250625 07:52:32.078918 2028 leader_election.cc:290] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.079126 1161 ts_tablet_manager.cc:1939] T 10c7f4b4187c428bbec136c85219da4d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:32.079913 986 leader_election.cc:343] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 445bfed532004108a12c8fd78370c1cb
I20250625 07:52:32.080438 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 10c7f4b4187c428bbec136c85219da4d (table default.Table_14bfc8ca6e024a2bba684efdcfc28fd3 [id=78145980babc49648859d7cbcc454242]) successfully deleted
W20250625 07:52:32.080659 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7c8abe36eab5498eb9bc3a0355101dd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c8abe36eab5498eb9bc3a0355101dd6 already in progress: creating tablet
W20250625 07:52:32.080912 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 308b387d366c4d7884898937a9fc7690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308b387d366c4d7884898937a9fc7690 already in progress: creating tablet
I20250625 07:52:32.080940 1162 tablet_bootstrap.cc:654] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:32.081117 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 08cd482ceec244c2bed2b116d5c14d89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cd482ceec244c2bed2b116d5c14d89 already in progress: creating tablet
W20250625 07:52:32.081432 985 leader_election.cc:343] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 445bfed532004108a12c8fd78370c1cb
I20250625 07:52:32.081529 985 leader_election.cc:304] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.081883 2028 raft_consensus.cc:2747] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.082391 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.083076 1158 tablet_bootstrap.cc:492] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.083249 1158 ts_tablet_manager.cc:1397] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20250625 07:52:32.083513 894 tablet_service.cc:1468] Processing CreateTablet for tablet 24c767b82ed24df1b145ff2df33ba155 (DEFAULT_TABLE table=default.Table_57bbb43218994668b3c285c21c088ffa [id=0b372231411b4a27bd7788e4fe589885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.083931 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c767b82ed24df1b145ff2df33ba155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.086627 1158 raft_consensus.cc:357] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.086845 1158 raft_consensus.cc:383] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.086958 1158 raft_consensus.cc:738] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.087230 1158 consensus_queue.cc:260] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.089462 1158 ts_tablet_manager.cc:1428] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:32.089565 876 tablet_service.cc:1468] Processing CreateTablet for tablet 46f81617a460458791e215b81c589241 (DEFAULT_TABLE table=default.Table_143a766c3eb149338d9b90575b6deaf1 [id=044d275c5fbb468db72885400493f0ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.089809 1158 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.089939 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f81617a460458791e215b81c589241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.092967 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.093335 890 tablet_service.cc:1468] Processing CreateTablet for tablet fd3ea59a079f4e05a2ee540dde05562b (DEFAULT_TABLE table=default.Table_0b3fc22526374bb1a70819a649be9dfe [id=c34ff2769412405d976aee0cfad52f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.093678 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd3ea59a079f4e05a2ee540dde05562b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.096947 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5099f01a4a124d23a7c32accbd576e54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.097270 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:32.097489 850 leader_election.cc:343] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.097584 850 leader_election.cc:304] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.097812 2112 raft_consensus.cc:2747] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.106956 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:32.108038 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1842b67d00e248c59e02c1603a8c5300 (DEFAULT_TABLE table=default.Table_04ba3f20d1184e0c9f9deb21ea849289 [id=2ff5265510874cacaf03166ff69e2469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.108410 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1842b67d00e248c59e02c1603a8c5300. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.113577 1163 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.113754 1163 ts_tablet_manager.cc:1397] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.010s
I20250625 07:52:32.114724 1163 raft_consensus.cc:357] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.114935 1163 raft_consensus.cc:383] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.114995 1163 raft_consensus.cc:738] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.115214 1163 consensus_queue.cc:260] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.115686 1163 ts_tablet_manager.cc:1428] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:32.115953 1163 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.117532 1163 tablet_bootstrap.cc:654] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.119390 1158 tablet_bootstrap.cc:654] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.120009 749 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0d45815ee45f3b48e4675f7ed901f (DEFAULT_TABLE table=default.Table_b6579bae985346f9bbb23891c04cbe8b [id=65e7da82cc5c4573afb74ef6d06e1f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.120507 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0d45815ee45f3b48e4675f7ed901f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:32.121469 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 484a49fab6134d059c64f5a3f2b29e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484a49fab6134d059c64f5a3f2b29e03 already in progress: creating tablet
I20250625 07:52:32.121973 893 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0d45815ee45f3b48e4675f7ed901f (DEFAULT_TABLE table=default.Table_b6579bae985346f9bbb23891c04cbe8b [id=65e7da82cc5c4573afb74ef6d06e1f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.122716 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0d45815ee45f3b48e4675f7ed901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.124333 2112 raft_consensus.cc:491] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.124459 2112 raft_consensus.cc:513] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.125150 2112 leader_election.cc:290] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.125669 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.126238 851 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.128573 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7386918a73e1456c8e9e59b4b24a27f4 (DEFAULT_TABLE table=default.Table_f30892ec994d427591925e7f321f3a4e [id=be7c54f60d164a9184ce26ee69c20383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.131141 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7386918a73e1456c8e9e59b4b24a27f4 (DEFAULT_TABLE table=default.Table_f30892ec994d427591925e7f321f3a4e [id=be7c54f60d164a9184ce26ee69c20383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.131731 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7386918a73e1456c8e9e59b4b24a27f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.134181 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7ab4243dccfa40bda777a9a70016ec4f" } new_table_name: "default.Table_92376fbb20ef47ef849e1ef6a99e3a17" modify_external_catalogs: true
I20250625 07:52:32.136198 596 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:32.138401 754 tablet_service.cc:1468] Processing CreateTablet for tablet c3f7ac725fd44e28bde713f5852acc89 (DEFAULT_TABLE table=default.Table_77374629d3b048f8a55d64335667dcea [id=89786bffabc8407597983077391a5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.138793 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f7ac725fd44e28bde713f5852acc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.138978 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c3f7ac725fd44e28bde713f5852acc89 (DEFAULT_TABLE table=default.Table_77374629d3b048f8a55d64335667dcea [id=89786bffabc8407597983077391a5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.139547 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.140492 752 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.140916 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbf323dfa034cc78904cb73077f4ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.142129 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5df78364c7b841979d418ddeecc476b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.142668 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5df78364c7b841979d418ddeecc476b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.144446 756 tablet_service.cc:1468] Processing CreateTablet for tablet f92ebe288b814b65ab327cabbd310c51 (DEFAULT_TABLE table=default.Table_d256e8aab118412d92953b648fe06283 [id=1fb6c7f9758e456eb124d3aa9dc5d885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.144896 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92ebe288b814b65ab327cabbd310c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.145143 1019 tablet_service.cc:1468] Processing CreateTablet for tablet f92ebe288b814b65ab327cabbd310c51 (DEFAULT_TABLE table=default.Table_d256e8aab118412d92953b648fe06283 [id=1fb6c7f9758e456eb124d3aa9dc5d885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.146000 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a44da8c8ff346118e917baceb47de8e" } new_table_name: "default.Table_bc09c435e0cd40328ca47a8a9b9c6c7c" modify_external_catalogs: true
I20250625 07:52:32.148945 885 tablet_service.cc:1468] Processing CreateTablet for tablet c3f7ac725fd44e28bde713f5852acc89 (DEFAULT_TABLE table=default.Table_77374629d3b048f8a55d64335667dcea [id=89786bffabc8407597983077391a5349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.149323 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f7ac725fd44e28bde713f5852acc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.156147 756 tablet_service.cc:1468] Processing CreateTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.156673 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74924e8dee943e0ae4c93f0db1d75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.157797 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.158454 891 tablet_service.cc:1468] Processing CreateTablet for tablet f92ebe288b814b65ab327cabbd310c51 (DEFAULT_TABLE table=default.Table_d256e8aab118412d92953b648fe06283 [id=1fb6c7f9758e456eb124d3aa9dc5d885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.158816 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92ebe288b814b65ab327cabbd310c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.160835 2045 raft_consensus.cc:491] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.161008 2045 raft_consensus.cc:513] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.161213 880 tablet_service.cc:1468] Processing CreateTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.161536 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74924e8dee943e0ae4c93f0db1d75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.161553 2045 leader_election.cc:290] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.172091 1163 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.172264 1163 ts_tablet_manager.cc:1397] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.003s
I20250625 07:52:32.174569 1163 raft_consensus.cc:357] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.174790 1163 raft_consensus.cc:383] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.174850 1163 raft_consensus.cc:738] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.175066 1163 consensus_queue.cc:260] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.175554 1163 ts_tablet_manager.cc:1428] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:32.175942 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 (DEFAULT_TABLE table=default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.177282 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:32.177358 1163 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.177459 914 raft_consensus.cc:2466] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:32.180500 713 leader_election.cc:304] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.180883 2045 raft_consensus.cc:2802] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:32.180977 2045 raft_consensus.cc:491] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:32.181049 2045 raft_consensus.cc:3058] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.181018 876 tablet_service.cc:1468] Processing CreateTablet for tablet 7386918a73e1456c8e9e59b4b24a27f4 (DEFAULT_TABLE table=default.Table_f30892ec994d427591925e7f321f3a4e [id=be7c54f60d164a9184ce26ee69c20383]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.181427 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7386918a73e1456c8e9e59b4b24a27f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.183115 1163 tablet_bootstrap.cc:654] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.183594 1013 tablet_service.cc:1468] Processing CreateTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 (DEFAULT_TABLE table=default.Table_2fa277660d76420a834714f10f39c3af [id=cd271faa9bca4ddfa1993f4b8a7ed937]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.183789 1158 tablet_bootstrap.cc:492] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.183897 1158 ts_tablet_manager.cc:1397] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.000s
I20250625 07:52:32.184016 2045 raft_consensus.cc:513] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.184607 2045 leader_election.cc:290] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.185194 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:32.185361 914 raft_consensus.cc:3058] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.187916 914 raft_consensus.cc:2466] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:32.189107 1163 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.190155 1163 ts_tablet_manager.cc:1397] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250625 07:52:32.190563 713 leader_election.cc:304] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.190833 2045 raft_consensus.cc:2802] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:32.190964 2045 raft_consensus.cc:695] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:32.191015 1163 raft_consensus.cc:357] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.191154 1163 raft_consensus.cc:383] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.191200 1163 raft_consensus.cc:738] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.191205 2045 consensus_queue.cc:237] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.191376 1163 consensus_queue.cc:260] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.191835 1163 ts_tablet_manager.cc:1428] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:32.192901 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.194584 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f92ebe288b814b65ab327cabbd310c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.194717 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:32.195175 850 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.195292 850 leader_election.cc:304] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.195463 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfea07d8441415e92d59f2f4e0b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.195554 2112 raft_consensus.cc:2747] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.198305 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7386918a73e1456c8e9e59b4b24a27f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.198770 1161 tablet_replica.cc:331] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.198957 1161 raft_consensus.cc:2241] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.199102 1161 raft_consensus.cc:2270] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.200212 1161 ts_tablet_manager.cc:1905] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.205312 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.205879 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74924e8dee943e0ae4c93f0db1d75f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.206139 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f7ac725fd44e28bde713f5852acc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.205168 1158 raft_consensus.cc:357] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.206547 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbf323dfa034cc78904cb73077f4ccf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:32.217811 713 leader_election.cc:343] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:32.218276 713 leader_election.cc:343] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:32.222720 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7be8ab5140aa4d48be978519cb99ed31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be8ab5140aa4d48be978519cb99ed31 already in progress: creating tablet
I20250625 07:52:32.206564 1158 raft_consensus.cc:383] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.223657 1158 raft_consensus.cc:738] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.224017 1158 consensus_queue.cc:260] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.226200 1158 ts_tablet_manager.cc:1428] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250625 07:52:32.232542 894 tablet_service.cc:1468] Processing CreateTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 (DEFAULT_TABLE table=default.Table_2fa277660d76420a834714f10f39c3af [id=cd271faa9bca4ddfa1993f4b8a7ed937]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.233004 749 tablet_service.cc:1468] Processing CreateTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 (DEFAULT_TABLE table=default.Table_2fa277660d76420a834714f10f39c3af [id=cd271faa9bca4ddfa1993f4b8a7ed937]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.233426 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfea07d8441415e92d59f2f4e0b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.234364 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.237180 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bd9a6cded85c4d6296f8468644de2016
I20250625 07:52:32.246037 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1e4d42ab40de4520ade73a572a5334dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.247648 2112 raft_consensus.cc:491] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.247799 2112 raft_consensus.cc:513] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.248414 2112 leader_election.cc:290] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.249035 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.249833 851 leader_election.cc:343] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.250172 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.251235 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e4d42ab40de4520ade73a572a5334dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:32.251536 850 leader_election.cc:343] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.251647 850 leader_election.cc:304] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.251909 2112 raft_consensus.cc:2747] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.252380 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9a6cded85c4d6296f8468644de2016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.253497 1158 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.253927 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0d45815ee45f3b48e4675f7ed901f (DEFAULT_TABLE table=default.Table_b6579bae985346f9bbb23891c04cbe8b [id=65e7da82cc5c4573afb74ef6d06e1f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:32.255198 1158 tablet_bootstrap.cc:654] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.255223 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0d45815ee45f3b48e4675f7ed901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.256096 890 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9a6cded85c4d6296f8468644de2016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.256474 756 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9a6cded85c4d6296f8468644de2016 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:32.257366 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bd9a6cded85c4d6296f8468644de2016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd9a6cded85c4d6296f8468644de2016 already in progress: creating tablet
I20250625 07:52:32.263854 1163 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.265689 1163 tablet_bootstrap.cc:654] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.268505 1989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=721 overwritten=0 applied=474 ignored=0} inserts{seen=443 ignored=0} mutations{seen=525 ignored=0} orphaned_commits=0. Pending: 247 replicates
I20250625 07:52:32.267863 1161 ts_tablet_manager.cc:1918] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.269716 1989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: Bootstrap complete.
I20250625 07:52:32.269304 1161 log.cc:1199] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/31942f0998bb4118a91f476325851d5c
I20250625 07:52:32.272657 1161 ts_tablet_manager.cc:1939] T 31942f0998bb4118a91f476325851d5c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.276204 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acfea07d8441415e92d59f2f4e0b72a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.276660 1158 tablet_bootstrap.cc:492] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.276877 1158 ts_tablet_manager.cc:1397] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250625 07:52:32.277701 1158 raft_consensus.cc:357] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.277959 1158 raft_consensus.cc:383] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.278064 1158 raft_consensus.cc:738] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.278369 1158 consensus_queue.cc:260] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.279331 1989 raft_consensus.cc:357] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Replica starting. Triggering 247 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:32.281486 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbf323dfa034cc78904cb73077f4ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:32.282686 1163 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.282860 1163 ts_tablet_manager.cc:1397] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20250625 07:52:32.283756 1163 raft_consensus.cc:357] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.283977 1163 raft_consensus.cc:383] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.284050 1163 raft_consensus.cc:738] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.284286 1163 consensus_queue.cc:260] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.289345 1158 ts_tablet_manager.cc:1428] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250625 07:52:32.291119 1163 ts_tablet_manager.cc:1428] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250625 07:52:32.291430 1163 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.295993 1158 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.297798 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 31942f0998bb4118a91f476325851d5c (table default.Table_a28f5f4927f34e03b2ea4784b65cfe43 [id=7686e223b6fb47e79ecb2257550a64b5]) successfully deleted
I20250625 07:52:32.299396 2045 raft_consensus.cc:491] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.299541 2045 raft_consensus.cc:513] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.300206 2045 leader_election.cc:290] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.300704 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:32.300889 914 raft_consensus.cc:2466] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:32.301458 713 leader_election.cc:304] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.301748 2045 raft_consensus.cc:2802] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:32.301846 2045 raft_consensus.cc:491] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:32.301911 2045 raft_consensus.cc:3058] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:32.302166 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 64ca83369ffe46ea9bd756f20da936cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64ca83369ffe46ea9bd756f20da936cf already in progress: creating tablet
I20250625 07:52:32.302796 1161 tablet_replica.cc:331] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.302976 1161 raft_consensus.cc:2241] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.303113 1161 raft_consensus.cc:2270] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.303777 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.304250 713 leader_election.cc:343] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.304458 2045 raft_consensus.cc:513] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.305008 2045 leader_election.cc:290] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.305408 2112 raft_consensus.cc:491] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.305539 2112 raft_consensus.cc:513] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.305574 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:32.305730 914 raft_consensus.cc:3058] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.306154 2112 leader_election.cc:290] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.306499 1163 tablet_bootstrap.cc:654] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.306726 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.307202 851 leader_election.cc:343] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.308344 914 raft_consensus.cc:2466] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:32.309080 713 leader_election.cc:304] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.309329 2045 raft_consensus.cc:2802] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:32.309494 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.309696 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.309829 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.309820 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.309945 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.310043 1166 tablet_replica.cc:331] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.310231 1166 raft_consensus.cc:2241] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.310374 1166 raft_consensus.cc:2270] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.310608 2045 raft_consensus.cc:695] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
W20250625 07:52:32.310971 850 leader_election.cc:343] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.310941 2045 consensus_queue.cc:237] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.311074 850 leader_election.cc:304] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.311273 2112 raft_consensus.cc:2747] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.311991 1166 ts_tablet_manager.cc:1905] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.317143 756 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.317178 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:32.317737 713 leader_election.cc:343] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.318171 1163 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.318354 1163 ts_tablet_manager.cc:1397] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250625 07:52:32.319231 1163 raft_consensus.cc:357] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.319389 1163 raft_consensus.cc:383] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.319453 1163 raft_consensus.cc:738] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.319692 1163 consensus_queue.cc:260] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.320290 1163 ts_tablet_manager.cc:1428] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:32.320538 1163 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.320533 1166 ts_tablet_manager.cc:1918] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.321338 1166 log.cc:1199] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/019959e9abed46fbbbb4755cb01cef44
I20250625 07:52:32.321718 1166 ts_tablet_manager.cc:1939] T 019959e9abed46fbbbb4755cb01cef44 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.322294 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 019959e9abed46fbbbb4755cb01cef44 (table default.Table_2c27be0b9107412facdd1bc4d768eebb [id=ea62e41d8d904cf5ac63b9d0a62cde09]) successfully deleted
I20250625 07:52:32.322737 1166 tablet_replica.cc:331] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.322975 1166 raft_consensus.cc:2241] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.323194 1166 raft_consensus.cc:2270] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.324117 1166 ts_tablet_manager.cc:1905] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.330670 1163 tablet_bootstrap.cc:654] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.341218 1158 tablet_bootstrap.cc:654] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.342954 1163 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.343113 1163 ts_tablet_manager.cc:1397] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250625 07:52:32.343339 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:32.343725 2045 raft_consensus.cc:491] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.343873 2045 raft_consensus.cc:513] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.343909 1163 raft_consensus.cc:357] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.344137 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.344316 756 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.344502 2045 leader_election.cc:290] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.345307 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.345866 713 leader_election.cc:343] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.345396 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:32.345989 1163 raft_consensus.cc:383] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.346066 914 raft_consensus.cc:2466] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:32.346068 1163 raft_consensus.cc:738] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.345613 890 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.346436 1163 consensus_queue.cc:260] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.346801 713 leader_election.cc:304] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.346928 1163 ts_tablet_manager.cc:1428] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:32.346856 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9c52eceef6e74723abcc7fce45c50f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.347368 2045 raft_consensus.cc:2802] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250625 07:52:32.347400 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c52eceef6e74723abcc7fce45c50f1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.347447 2045 raft_consensus.cc:491] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:32.347505 2045 raft_consensus.cc:3058] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.349318 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 07a07af928ff4d3d894b074ef0cf44eb from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:32.350109 2045 raft_consensus.cc:513] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.350220 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b36ed46abbe249a897381bd2b81d9d12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.350641 2045 leader_election.cc:290] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:32.350677 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b36ed46abbe249a897381bd2b81d9d12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.351464 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:32.351517 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cadd41655404316b10939e46d370c76" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:32.351671 914 raft_consensus.cc:3058] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:32.351954 713 leader_election.cc:343] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.352663 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4f5e9909515b4b28b92b7ece88d91b4b" } new_table_name: "default.Table_deb382f48ba949ed979b457ddbd62fc1" modify_external_catalogs: true
I20250625 07:52:32.355022 914 raft_consensus.cc:2466] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:32.352945 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8f25febb41c403ca8be268206157756 consensus state has no leader
I20250625 07:52:32.355515 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.355907 713 leader_election.cc:304] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.355957 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0ead4f99381840579eabcd8334f44489"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.356447 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ead4f99381840579eabcd8334f44489 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.358256 1161 ts_tablet_manager.cc:1905] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.358529 2112 raft_consensus.cc:491] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.358678 2112 raft_consensus.cc:513] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.359264 2112 leader_election.cc:290] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.359737 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "110fe299c7c44df69f459da72a1138a5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.359851 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "110fe299c7c44df69f459da72a1138a5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.360183 850 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:32.360394 851 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.360478 851 leader_election.cc:304] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.364794 2112 raft_consensus.cc:2747] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.366977 592 catalog_manager.cc:5582] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:32.367530 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:32.367731 2045 raft_consensus.cc:2802] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:32.367940 592 catalog_manager.cc:5582] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:32.368189 2045 raft_consensus.cc:695] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:32.368569 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 4b4627f4da6b4542aff1e44157087b14 (table default.Table_d5657e03390941c28681972134ee0159 [id=618d684f882f425dab0d9a7395deb004]). Expected version 3 got 0
I20250625 07:52:32.368736 2045 consensus_queue.cc:237] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.369823 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:32.370802 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9f2a23f9d8f5431f9835acf040a4a5e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.371546 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f2a23f9d8f5431f9835acf040a4a5e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.354153 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c250beeb533e41af826edcc10557bf9a" } new_table_name: "default.Table_5b63803e7b284716bdf7d2ccae0c95fa" modify_external_catalogs: true
I20250625 07:52:32.377121 1793 raft_consensus.cc:491] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.377267 1793 raft_consensus.cc:513] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.377868 1793 leader_election.cc:290] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.378662 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfcef152c9844a59989c532bd60c07d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.379278 851 leader_election.cc:343] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.380282 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfcef152c9844a59989c532bd60c07d" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.381024 850 leader_election.cc:343] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.381143 850 leader_election.cc:304] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.381778 1793 raft_consensus.cc:2747] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.381923 1793 raft_consensus.cc:491] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.382006 1793 raft_consensus.cc:513] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.382501 1793 leader_election.cc:290] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.383095 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e5a207d515449e298936d8bb532437a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.383534 851 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.384117 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e5a207d515449e298936d8bb532437a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.384542 850 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.384634 850 leader_election.cc:304] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.385056 2112 raft_consensus.cc:2747] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.386478 1010 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47290 (request call id 144) took 1097 ms. Trace:
I20250625 07:52:32.418828 1160 ts_tablet_manager.cc:1918] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.418974 1160 log.cc:1199] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/f04016cda7f746d49e30be3b1c9c18b6
I20250625 07:52:32.419308 1160 ts_tablet_manager.cc:1939] T f04016cda7f746d49e30be3b1c9c18b6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:32.421855 1989 raft_consensus.cc:738] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a3faeffaa4422ea73b98ee74768a4a, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.425258 1163 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.427002 1163 tablet_bootstrap.cc:654] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.427229 1989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 474, Last appended: 1.721, Last appended by leader: 721, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:32.434823 1162 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.435133 1162 ts_tablet_manager.cc:1397] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.372s user 0.026s sys 0.045s
I20250625 07:52:32.436153 1162 raft_consensus.cc:357] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.436514 1162 raft_consensus.cc:383] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.436657 1162 raft_consensus.cc:738] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.437040 1162 consensus_queue.cc:260] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.437845 1162 ts_tablet_manager.cc:1428] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:32.439211 1163 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.439399 1163 ts_tablet_manager.cc:1397] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20250625 07:52:32.440351 1163 raft_consensus.cc:357] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.440608 1163 raft_consensus.cc:383] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.440749 1163 raft_consensus.cc:738] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.441020 1163 consensus_queue.cc:260] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.441687 1163 ts_tablet_manager.cc:1428] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:32.442006 1163 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.443058 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
W20250625 07:52:32.443324 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4718c51c79ac41cebfca7dcc567ddecd consensus state has no leader
I20250625 07:52:32.444057 1163 tablet_bootstrap.cc:654] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.446022 1162 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.447573 1162 tablet_bootstrap.cc:654] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.449226 1163 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.449421 1163 ts_tablet_manager.cc:1397] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20250625 07:52:32.450316 1163 raft_consensus.cc:357] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.450543 1163 raft_consensus.cc:383] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.450663 1163 raft_consensus.cc:738] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.450785 2131 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:32.450946 1163 consensus_queue.cc:260] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.451392 1163 ts_tablet_manager.cc:1428] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:32.451670 1163 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.452848 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0712d103f5094acb959ac7e976036190"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.453285 1163 tablet_bootstrap.cc:654] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:32.453401 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0712d103f5094acb959ac7e976036190 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.455098 1162 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.455277 1162 ts_tablet_manager.cc:1397] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20250625 07:52:32.456015 1162 raft_consensus.cc:357] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.456231 1162 raft_consensus.cc:383] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.456327 1162 raft_consensus.cc:738] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.456606 1162 consensus_queue.cc:260] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.456758 890 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.457154 1162 ts_tablet_manager.cc:1428] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:32.457327 890 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.457386 1162 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:32.458279 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99421eff752142348c8a899e7ac961b1 consensus state has no leader
I20250625 07:52:32.458671 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:32.459045 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f07796775e9f4e0299fa92332951f3f5" } modify_external_catalogs: true
I20250625 07:52:32.459255 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f07796775e9f4e0299fa92332951f3f5" } modify_external_catalogs: true
I20250625 07:52:32.459551 1162 tablet_bootstrap.cc:654] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.460314 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:32.460515 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ff871a8a9993429da4d087d0362c3ced" } modify_external_catalogs: true
I20250625 07:52:32.460734 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ff871a8a9993429da4d087d0362c3ced" } modify_external_catalogs: true
I20250625 07:52:32.457867 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3f799c3e479a406690670e68e70295cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.462548 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f799c3e479a406690670e68e70295cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.467249 1162 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.467439 1162 ts_tablet_manager.cc:1397] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20250625 07:52:32.468140 1162 raft_consensus.cc:357] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.468365 1162 raft_consensus.cc:383] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.468452 1162 raft_consensus.cc:738] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.468732 1162 consensus_queue.cc:260] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.469698 1989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:52:32.473157 1162 ts_tablet_manager.cc:1428] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 07:52:32.469005 1166 ts_tablet_manager.cc:1918] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.473672 1166 log.cc:1199] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:32.473695 756 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.474314 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.387905 1158 tablet_bootstrap.cc:492] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.476769 1010 rpcz_store.cc:276] 0625 07:52:31.289191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.289573 (+ 382us) service_pool.cc:225] Handling call
0625 07:52:32.386452 (+1096879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250625 07:52:32.476892 1158 ts_tablet_manager.cc:1397] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.000s
I20250625 07:52:32.388727 2028 raft_consensus.cc:491] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.477172 2028 raft_consensus.cc:513] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.477651 1158 raft_consensus.cc:357] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.477797 1158 raft_consensus.cc:383] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.477857 1158 raft_consensus.cc:738] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.478015 1158 consensus_queue.cc:260] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.449088 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2db9e8d0fefa47bfaeee462dba096b49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:32.480460 986 leader_election.cc:343] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9418b81f010d4f369d3409e9364bc5fa
W20250625 07:52:32.481230 985 leader_election.cc:343] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9418b81f010d4f369d3409e9364bc5fa
I20250625 07:52:32.481361 985 leader_election.cc:304] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.477807 2028 leader_election.cc:290] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:32.482096 2028 raft_consensus.cc:2747] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.485363 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1060) took 1628 ms. Trace:
I20250625 07:52:32.485471 1160 rpcz_store.cc:276] 0625 07:52:30.857295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.857338 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:32.011287 (+1153949us) ts_tablet_manager.cc:1145] Deleting tablet f04016cda7f746d49e30be3b1c9c18b6
0625 07:52:32.011299 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.367410 (+356111us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.418713 (+ 51303us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.485338 (+ 66625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1153809}
I20250625 07:52:32.485935 1160 tablet_replica.cc:331] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:32.486121 1160 raft_consensus.cc:2241] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.486274 1160 raft_consensus.cc:2270] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.486903 1160 ts_tablet_manager.cc:1905] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.487322 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet f04016cda7f746d49e30be3b1c9c18b6 (table default.Table_634beccfc1a34257bd0bc04d1d020530 [id=989d9b62801c467e8bba2e03c0f13395]) successfully deleted
I20250625 07:52:32.457546 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ca0144e3c5142ac9fcbd927b7585a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.493196 2028 raft_consensus.cc:491] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.493372 2028 raft_consensus.cc:513] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.494017 2028 leader_election.cc:290] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:32.495254 986 leader_election.cc:343] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:32.500159 1163 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.500581 1163 ts_tablet_manager.cc:1397] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.004s
I20250625 07:52:32.474028 1162 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.501920 1163 raft_consensus.cc:357] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.503403 1163 raft_consensus.cc:383] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.504891 1163 raft_consensus.cc:738] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.505375 1163 consensus_queue.cc:260] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.506264 1166 ts_tablet_manager.cc:1939] T a1baf94a051e485eaf4c0e074e147252 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.506359 1163 ts_tablet_manager.cc:1428] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:32.506742 1163 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.506268 1850 raft_consensus.cc:491] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:32.506876 985 leader_election.cc:343] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:32.506995 985 leader_election.cc:304] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.507226 1850 raft_consensus.cc:513] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.507428 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a1baf94a051e485eaf4c0e074e147252 (table default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]) successfully deleted
I20250625 07:52:32.507827 1850 leader_election.cc:290] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.508091 2178 raft_consensus.cc:2747] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:32.508740 713 leader_election.cc:343] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:32.509452 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c50c35cac8e94e088cdacb664b8adfe1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.507335 1166 tablet_replica.cc:331] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.512344 1166 raft_consensus.cc:2241] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:32.512734 713 leader_election.cc:343] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.512887 713 leader_election.cc:304] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.512768 1162 tablet_bootstrap.cc:654] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.513124 2161 raft_consensus.cc:2747] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.513386 2178 raft_consensus.cc:491] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.513558 2178 raft_consensus.cc:513] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.513983 1166 raft_consensus.cc:2270] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.514626 2178 leader_election.cc:290] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:32.515192 986 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3ca0144e3c5142ac9fcbd927b7585a3c
W20250625 07:52:32.515447 985 leader_election.cc:343] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:32.515556 985 leader_election.cc:304] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.515722 1166 ts_tablet_manager.cc:1905] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:32.518620 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f7e4e88633c42a7b1bf70a4c64cea67 consensus state has no leader
I20250625 07:52:32.520767 2178 raft_consensus.cc:2747] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.521097 1158 ts_tablet_manager.cc:1428] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.044s user 0.003s sys 0.003s
I20250625 07:52:32.523808 1158 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.525280 1158 tablet_bootstrap.cc:654] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.530263 1162 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.530589 1162 ts_tablet_manager.cc:1397] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.003s
I20250625 07:52:32.531308 1162 raft_consensus.cc:357] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.531585 1162 raft_consensus.cc:383] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.531684 1162 raft_consensus.cc:738] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.532212 1162 consensus_queue.cc:260] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.532801 1162 ts_tablet_manager.cc:1428] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:32.533125 1162 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.533138 1048 raft_consensus.cc:3058] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.534655 1162 tablet_bootstrap.cc:654] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.535063 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_50c078cf2e4148df96289ccba9f2da2d" } modify_external_catalogs: true
I20250625 07:52:32.535296 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_50c078cf2e4148df96289ccba9f2da2d" } modify_external_catalogs: true
I20250625 07:52:32.536253 2112 raft_consensus.cc:491] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.536403 2112 raft_consensus.cc:513] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.537010 2112 leader_election.cc:290] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.541317 2112 raft_consensus.cc:491] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.541505 2112 raft_consensus.cc:513] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.541749 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0cc3ba09d5e4c3d88e74fffc97eea8b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.542137 2112 leader_election.cc:290] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.542708 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet a33b7cab19244ec08d3821b6cc4c072d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:32.543378 850 leader_election.cc:343] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.543491 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.543954 850 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.532797 2171 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:32.541486 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0cc3ba09d5e4c3d88e74fffc97eea8b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:32.542801 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:32.468484 1161 ts_tablet_manager.cc:1918] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.548736 1161 log.cc:1199] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/08bb47266e8b4a0fa5ec613560b8549e
W20250625 07:52:32.548877 851 leader_election.cc:343] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.549014 851 leader_election.cc:304] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.549328 1161 ts_tablet_manager.cc:1939] T 08bb47266e8b4a0fa5ec613560b8549e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:32.549410 851 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.549497 851 leader_election.cc:304] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.549770 1161 tablet_replica.cc:331] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.549952 1161 raft_consensus.cc:2241] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.550096 1161 raft_consensus.cc:2270] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.550272 1793 raft_consensus.cc:2747] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.550428 1793 raft_consensus.cc:2747] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.550627 1161 ts_tablet_manager.cc:1905] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.550906 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 08bb47266e8b4a0fa5ec613560b8549e (table default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38 [id=3ee7ec23fdfd40e68b9f53bf74a4f843]) successfully deleted
I20250625 07:52:32.559548 1158 tablet_bootstrap.cc:492] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.559681 1158 ts_tablet_manager.cc:1397] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250625 07:52:32.560400 1158 raft_consensus.cc:357] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.560560 1158 raft_consensus.cc:383] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.560606 1158 raft_consensus.cc:738] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.563519 1158 consensus_queue.cc:260] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.564455 1158 ts_tablet_manager.cc:1428] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:32.564766 1158 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.566320 1158 tablet_bootstrap.cc:654] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.566593 1166 ts_tablet_manager.cc:1918] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.566711 1166 log.cc:1199] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0929ed76622248d1a41f08f4258108cd
I20250625 07:52:32.567008 1166 ts_tablet_manager.cc:1939] T 0929ed76622248d1a41f08f4258108cd P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.567492 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4e2dc27726d944c5885d62ed4258acfc" } modify_external_catalogs: true
I20250625 07:52:32.567807 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4e2dc27726d944c5885d62ed4258acfc" } modify_external_catalogs: true
I20250625 07:52:32.568002 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0929ed76622248d1a41f08f4258108cd (table default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]) successfully deleted
I20250625 07:52:32.575699 1166 tablet_replica.cc:331] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.576321 1166 raft_consensus.cc:2241] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.576344 1162 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.576975 1162 ts_tablet_manager.cc:1397] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250625 07:52:32.577246 915 raft_consensus.cc:1273] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:32.577675 1162 raft_consensus.cc:357] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.577837 1162 raft_consensus.cc:383] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.577886 1162 raft_consensus.cc:738] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.578094 1162 consensus_queue.cc:260] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.578464 1850 consensus_queue.cc:1035] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:32.578570 1162 ts_tablet_manager.cc:1428] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 07:52:32.578812 1162 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.580305 1162 tablet_bootstrap.cc:654] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.581947 1166 raft_consensus.cc:2270] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.582657 1166 ts_tablet_manager.cc:1905] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:32.585702 713 consensus_peers.cc:487] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:32.588574 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3197b8d0402490b86e0fbc10c91c62e" } modify_external_catalogs: true
I20250625 07:52:32.588766 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3197b8d0402490b86e0fbc10c91c62e" } modify_external_catalogs: true
I20250625 07:52:32.589064 1158 tablet_bootstrap.cc:492] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.589632 1158 ts_tablet_manager.cc:1397] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.001s
I20250625 07:52:32.590734 1158 raft_consensus.cc:357] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.590895 1158 raft_consensus.cc:383] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.590938 1158 raft_consensus.cc:738] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.595273 1162 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.595463 1162 ts_tablet_manager.cc:1397] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20250625 07:52:32.596513 1158 consensus_queue.cc:260] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.597489 1158 ts_tablet_manager.cc:1428] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 07:52:32.597765 1158 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.598003 1162 raft_consensus.cc:357] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.598232 1162 raft_consensus.cc:383] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.598327 1162 raft_consensus.cc:738] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.598615 1162 consensus_queue.cc:260] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.599419 1162 ts_tablet_manager.cc:1428] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:32.599965 1162 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.601943 1162 tablet_bootstrap.cc:654] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.601897 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7d69398f95854f01909ebfe2310d23f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.602466 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d69398f95854f01909ebfe2310d23f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.602718 1158 tablet_bootstrap.cc:654] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.603729 1166 ts_tablet_manager.cc:1918] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.604830 1166 log.cc:1199] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:32.605268 1166 ts_tablet_manager.cc:1939] T e289ea5553ac4ff983871ece4fc72ee5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.606411 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e289ea5553ac4ff983871ece4fc72ee5 (table default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]) successfully deleted
I20250625 07:52:32.607348 1166 tablet_replica.cc:331] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.607481 1158 tablet_bootstrap.cc:492] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.607635 1158 ts_tablet_manager.cc:1397] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250625 07:52:32.607728 1166 raft_consensus.cc:2241] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.607925 1166 raft_consensus.cc:2270] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.608346 1158 raft_consensus.cc:357] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.608506 1158 raft_consensus.cc:383] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.608562 1158 raft_consensus.cc:738] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.608845 1166 ts_tablet_manager.cc:1905] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.608767 1158 consensus_queue.cc:260] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.611400 1158 ts_tablet_manager.cc:1428] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:32.611682 1158 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:32.611945 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4718c51c79ac41cebfca7dcc567ddecd (table default.Table_bc09c435e0cd40328ca47a8a9b9c6c7c [id=8f05b151cc6d4b6ba0c5670a94a3e8d8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4718c51c79ac41cebfca7dcc567ddecd consensus state has no leader
I20250625 07:52:32.613669 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d6623a19ca9c41db9ea86fb93a800ce4" } new_table_name: "default.Table_b16a97fd65d14358a0b7bc51c14f5f61" modify_external_catalogs: true
I20250625 07:52:32.618469 1158 tablet_bootstrap.cc:654] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.619195 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f2ee3980ed9747af87329fd3a1dfd699
I20250625 07:52:32.621484 752 tablet_service.cc:1515] Processing DeleteTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.622550 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.623701 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f2ee3980ed9747af87329fd3a1dfd699 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.624452 1162 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.624651 1162 ts_tablet_manager.cc:1397] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20250625 07:52:32.625481 1162 raft_consensus.cc:357] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.625725 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:32.625725 1162 raft_consensus.cc:383] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.625792 1162 raft_consensus.cc:738] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.626024 1162 consensus_queue.cc:260] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.628085 1162 ts_tablet_manager.cc:1428] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:32.628389 1162 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.630013 1162 tablet_bootstrap.cc:654] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.630479 2112 raft_consensus.cc:491] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.630604 2112 raft_consensus.cc:513] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.631155 2112 leader_election.cc:290] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.632164 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_49db16d175f84573b2f9d34ce29ac535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.632385 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97b9a14151fc472e956126620633be32" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.632767 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49db16d175f84573b2f9d34ce29ac535 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.633889 1158 tablet_bootstrap.cc:492] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
W20250625 07:52:32.633919 850 leader_election.cc:343] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.634003 1158 ts_tablet_manager.cc:1397] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.008s
I20250625 07:52:32.634671 1158 raft_consensus.cc:357] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.634825 1158 raft_consensus.cc:383] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.634871 1158 raft_consensus.cc:738] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.631992 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97b9a14151fc472e956126620633be32" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:32.635052 1158 consensus_queue.cc:260] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.635213 2171 tablet.cc:1722] T c5fe3ea5745743019d23d49ff808070a P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 07:52:32.635401 851 leader_election.cc:343] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.635494 851 leader_election.cc:304] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.635835 2112 raft_consensus.cc:2747] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.638650 1166 ts_tablet_manager.cc:1918] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.640241 1158 ts_tablet_manager.cc:1428] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:32.640412 1166 log.cc:1199] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:32.640542 1158 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.640921 1166 ts_tablet_manager.cc:1939] T c50c35cac8e94e088cdacb664b8adfe1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.644652 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c50c35cac8e94e088cdacb664b8adfe1 (table default.Table_f6e3be4354c248d8abb373c9f0b5c6c9 [id=860ff357dc3a4a2dad9d850c113cf464]) successfully deleted
W20250625 07:52:32.645500 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet af8c114c1c464bcc95dbd287c88976d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af8c114c1c464bcc95dbd287c88976d6 already in progress: creating tablet
I20250625 07:52:32.645895 1166 tablet_replica.cc:331] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.645833 2131 tablet.cc:1722] T c5fe3ea5745743019d23d49ff808070a P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:32.646301 1166 raft_consensus.cc:2241] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.646450 1166 raft_consensus.cc:2270] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.646945 1166 ts_tablet_manager.cc:1905] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.647980 1158 tablet_bootstrap.cc:654] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.651834 1162 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.651986 1162 ts_tablet_manager.cc:1397] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250625 07:52:32.653159 1162 raft_consensus.cc:357] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.653371 1162 raft_consensus.cc:383] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.653420 1162 raft_consensus.cc:738] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.653662 1162 consensus_queue.cc:260] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.654395 1162 ts_tablet_manager.cc:1428] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:32.655673 1162 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.656181 1158 tablet_bootstrap.cc:492] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.656319 1158 ts_tablet_manager.cc:1397] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20250625 07:52:32.657114 1158 raft_consensus.cc:357] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.657276 1158 raft_consensus.cc:383] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.657338 1158 raft_consensus.cc:738] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.657541 1158 consensus_queue.cc:260] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.661453 1162 tablet_bootstrap.cc:654] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.661876 1158 ts_tablet_manager.cc:1428] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:32.662336 1158 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.664870 1158 tablet_bootstrap.cc:654] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.667871 1166 ts_tablet_manager.cc:1918] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.668213 1166 log.cc:1199] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:32.668743 1166 ts_tablet_manager.cc:1939] T ccddd630a82341cba7a857e67a349fe0 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.670645 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet ccddd630a82341cba7a857e67a349fe0 (table default.Table_80cbf681213f4716b1b7fd920d71b59c [id=3a15e8259a5f4b1283f0c106d55baff1]) successfully deleted
I20250625 07:52:32.675532 1166 tablet_replica.cc:331] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:32.676055 1166 raft_consensus.cc:2241] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.676060 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:32.676292 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 89c703dc3249408faeb522cef0248111 (table default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353]). Expected version 1 got 0
I20250625 07:52:32.676934 1158 tablet_bootstrap.cc:492] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.677192 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_033160d4bb6d460093d51e5969e61194" } new_table_name: "default.Table_be186691a36341219d1f1a882b27e001" modify_external_catalogs: true
I20250625 07:52:32.677062 1158 ts_tablet_manager.cc:1397] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20250625 07:52:32.676971 1166 raft_consensus.cc:2270] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.677827 1166 ts_tablet_manager.cc:1905] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.678170 1158 raft_consensus.cc:357] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.678313 1158 raft_consensus.cc:383] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.678365 1158 raft_consensus.cc:738] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.678531 1158 consensus_queue.cc:260] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.681985 1158 ts_tablet_manager.cc:1428] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 07:52:32.682231 1158 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.683815 1158 tablet_bootstrap.cc:654] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.686607 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_92376fbb20ef47ef849e1ef6a99e3a17 [id=0e236158e41f4ea5aef616e7777bb94f] alter complete (version 5)
I20250625 07:52:32.691668 1163 tablet_bootstrap.cc:654] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.691990 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.692128 1166 ts_tablet_manager.cc:1918] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.692363 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.695068 1166 log.cc:1199] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/871b603cb92249c6b0f16c825bf46232
I20250625 07:52:32.695734 1166 ts_tablet_manager.cc:1939] T 871b603cb92249c6b0f16c825bf46232 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:32.697607 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 871b603cb92249c6b0f16c825bf46232 (table default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]) successfully deleted
W20250625 07:52:32.698586 713 consensus_peers.cc:487] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:32.699501 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:32.699841 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
W20250625 07:52:32.699901 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f2ee3980ed9747af87329fd3a1dfd699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ee3980ed9747af87329fd3a1dfd699 already in progress: creating tablet
I20250625 07:52:32.700073 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 1228018baf3340d99b3f94b150d0d533 (table default.Table_92376fbb20ef47ef849e1ef6a99e3a17 [id=0e236158e41f4ea5aef616e7777bb94f]). Expected version 5 got 0
I20250625 07:52:32.701857 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 29f2997d1d714100950383e6af04f315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.707945 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8a899790713d4646a850971bc3aebd9f
I20250625 07:52:32.709447 1161 ts_tablet_manager.cc:1918] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.711001 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a899790713d4646a850971bc3aebd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:32.711054 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a899790713d4646a850971bc3aebd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:32.712010 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8a899790713d4646a850971bc3aebd9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a899790713d4646a850971bc3aebd9f already in progress: creating tablet
I20250625 07:52:32.721859 1160 ts_tablet_manager.cc:1918] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.721992 1160 log.cc:1199] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/f4bb74391a1c4e2792226d259a07f977
I20250625 07:52:32.722306 1160 ts_tablet_manager.cc:1939] T f4bb74391a1c4e2792226d259a07f977 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:32.723070 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1061) took 1865 ms. Trace:
I20250625 07:52:32.723167 1160 rpcz_store.cc:276] 0625 07:52:30.857576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.857614 (+ 38us) service_pool.cc:225] Handling call
0625 07:52:32.485883 (+1628269us) ts_tablet_manager.cc:1145] Deleting tablet f4bb74391a1c4e2792226d259a07f977
0625 07:52:32.485895 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.624962 (+139067us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.721727 (+ 96765us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.723048 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628119}
I20250625 07:52:32.723599 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet f4bb74391a1c4e2792226d259a07f977 (table default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]) successfully deleted
I20250625 07:52:32.723991 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1062) took 1865 ms. Trace:
I20250625 07:52:32.724103 1160 rpcz_store.cc:276] 0625 07:52:30.858093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.858133 (+ 40us) service_pool.cc:225] Handling call
0625 07:52:32.723874 (+1865741us) ts_tablet_manager.cc:1145] Deleting tablet 683954dc7ecc4185847f46a05fd2c53d
0625 07:52:32.723884 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.723976 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1865605}
W20250625 07:52:32.724511 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 683954dc7ecc4185847f46a05fd2c53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683954dc7ecc4185847f46a05fd2c53d already in progress: creating tablet
I20250625 07:52:32.724880 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1063) took 1866 ms. Trace:
I20250625 07:52:32.711866 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a899790713d4646a850971bc3aebd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.716585 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 494cacf2dcbb40d3b2c9f05f64809870 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.730376 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8d34bfe377ba421daede5ae68848a2d4
I20250625 07:52:32.732437 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:32.733289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8d34bfe377ba421daede5ae68848a2d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d34bfe377ba421daede5ae68848a2d4 already in progress: creating tablet
I20250625 07:52:32.734138 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.740564 1161 log.cc:1199] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a0518fbc7f52454296ae79a3c92ef800
I20250625 07:52:32.741096 1161 ts_tablet_manager.cc:1939] T a0518fbc7f52454296ae79a3c92ef800 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.741747 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a0518fbc7f52454296ae79a3c92ef800 (table default.Table_6e28f36e9e034b489bf92ac42b5f0e28 [id=8fe2061799bb4310b8d399b7c372173d]) successfully deleted
I20250625 07:52:32.742094 1161 tablet_replica.cc:331] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.742156 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c261d05cf03b422fa40dc4ffdd2fb691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.742312 1161 raft_consensus.cc:2241] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.742460 1161 raft_consensus.cc:2270] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.743011 1161 ts_tablet_manager.cc:1905] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:32.744100 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c261d05cf03b422fa40dc4ffdd2fb691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.753636 2178 raft_consensus.cc:491] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.753854 2178 raft_consensus.cc:513] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.754667 2178 leader_election.cc:290] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:32.755882 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_404e6f3b9acf49a18305dbe0b2cac534" } new_table_name: "default.Table_bb0f6a2693cc429fbb2293b5ba54b563" modify_external_catalogs: true
W20250625 07:52:32.755957 986 leader_election.cc:343] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 250175f54ba94a9fbc2455d9cf27fb2e
W20250625 07:52:32.756735 985 leader_election.cc:343] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:32.756875 985 leader_election.cc:304] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.757313 2178 raft_consensus.cc:2747] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:32.761785 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8ab63bf0a0f4be29cb027505cedf835 consensus state has no leader
I20250625 07:52:32.771752 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_59367713e2b8459791d1db51abc86d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.772347 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59367713e2b8459791d1db51abc86d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.774019 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_82cd3da25b72491089bf529b94aafabd" } new_table_name: "default.Table_c6761e13384642f19befe5477a330593" modify_external_catalogs: true
I20250625 07:52:32.776077 2162 raft_consensus.cc:491] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.776226 2162 raft_consensus.cc:513] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.776968 2162 leader_election.cc:290] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.777776 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2bb060cdd9a48688967989d7491cfc1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.778309 713 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b2bb060cdd9a48688967989d7491cfc1
W20250625 07:52:32.778713 713 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.778815 713 leader_election.cc:304] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.779189 2162 raft_consensus.cc:2747] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.793430 595 master_service.cc:948] ReplaceTablet: received request to replace tablet b8b5bb3fe3ef4bfdba93b862784a9f61 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:32.729873 1163 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.734838 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d34bfe377ba421daede5ae68848a2d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:32.796618 1163 ts_tablet_manager.cc:1397] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.290s user 0.004s sys 0.008s
I20250625 07:52:32.797427 1163 raft_consensus.cc:357] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.797592 1163 raft_consensus.cc:383] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.797634 1163 raft_consensus.cc:738] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.797894 1163 consensus_queue.cc:260] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.798399 1163 ts_tablet_manager.cc:1428] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:32.800046 2112 raft_consensus.cc:491] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.800159 2112 raft_consensus.cc:513] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.796502 1160 rpcz_store.cc:276] 0625 07:52:30.858499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.858539 (+ 40us) service_pool.cc:225] Handling call
0625 07:52:32.724780 (+1866241us) ts_tablet_manager.cc:1145] Deleting tablet 652e07d46af74945bf504f7d9015a446
0625 07:52:32.724789 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.724866 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866116}
I20250625 07:52:32.800660 2112 leader_election.cc:290] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:32.800676 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 652e07d46af74945bf504f7d9015a446 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 652e07d46af74945bf504f7d9015a446 already in progress: creating tablet
I20250625 07:52:32.801370 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdad7dc1b27e4d87a747642a6d47230c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.801466 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fdad7dc1b27e4d87a747642a6d47230c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.802060 851 leader_election.cc:343] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:32.802310 850 leader_election.cc:343] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.802402 850 leader_election.cc:304] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.802771 2112 raft_consensus.cc:2747] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.804088 1162 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.804320 1162 ts_tablet_manager.cc:1397] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.149s user 0.022s sys 0.051s
I20250625 07:52:32.806365 1162 raft_consensus.cc:357] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.806645 1162 raft_consensus.cc:383] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.806771 1162 raft_consensus.cc:738] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.807066 1162 consensus_queue.cc:260] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:32.807765 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7df088d0ba6f4b1888aa48400a2b7fa3 consensus state has no leader
I20250625 07:52:32.809337 2112 raft_consensus.cc:491] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.809476 2112 raft_consensus.cc:513] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.810024 2112 leader_election.cc:290] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.810374 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0657163b9a0d4f7f99f57f7fd1f32462" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:32.810842 851 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.811241 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0657163b9a0d4f7f99f57f7fd1f32462" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:32.811893 850 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.811988 850 leader_election.cc:304] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.812729 2112 raft_consensus.cc:2747] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.814090 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a5179ed8694740ada7407f92a210c71a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:32.814620 1162 ts_tablet_manager.cc:1428] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250625 07:52:32.814630 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5179ed8694740ada7407f92a210c71a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.815021 1162 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.819438 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:32.825718 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9b46e6d83dcd438fb7a25fb360da7435" } new_table_name: "default.Table_9b08172057b049b4b6e9f459a8300dba" modify_external_catalogs: true
I20250625 07:52:32.825824 2161 consensus_queue.cc:1035] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:32.828650 1162 tablet_bootstrap.cc:654] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:32.832430 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8fe5a9156b14d1ea79fd1f3a04dcd20 consensus state has no leader
I20250625 07:52:32.835116 1161 ts_tablet_manager.cc:1918] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.837265 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4e553aa8d6d4472ba0f5f9124ff6f580"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.837774 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e553aa8d6d4472ba0f5f9124ff6f580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.838073 2162 raft_consensus.cc:491] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.838186 2162 raft_consensus.cc:513] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.838723 2162 leader_election.cc:290] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.839630 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:32.839804 914 raft_consensus.cc:2466] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:32.840260 713 leader_election.cc:304] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.840384 1163 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.840615 2162 raft_consensus.cc:2802] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:32.840723 2162 raft_consensus.cc:491] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:32.840793 2162 raft_consensus.cc:3058] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.842237 1163 tablet_bootstrap.cc:654] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.843092 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1064) took 1984 ms. Trace:
I20250625 07:52:32.843164 1160 rpcz_store.cc:276] 0625 07:52:30.858825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.858953 (+ 128us) service_pool.cc:225] Handling call
0625 07:52:32.801127 (+1942174us) ts_tablet_manager.cc:1145] Deleting tablet 78a1bfd87f084d50bd2b602b89c18a2f
0625 07:52:32.842974 (+ 41847us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.843075 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942040}
I20250625 07:52:32.843664 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1066) took 1983 ms. Trace:
I20250625 07:52:32.843732 1160 rpcz_store.cc:276] 0625 07:52:30.860791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.860836 (+ 45us) service_pool.cc:225] Handling call
0625 07:52:32.843568 (+1982732us) ts_tablet_manager.cc:1145] Deleting tablet b782414a682246feb2811b4e3bf8b5a2
0625 07:52:32.843576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.843651 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982587}
W20250625 07:52:32.843762 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 78a1bfd87f084d50bd2b602b89c18a2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a1bfd87f084d50bd2b602b89c18a2f already in progress: creating tablet
I20250625 07:52:32.843984 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1068) took 1975 ms. Trace:
I20250625 07:52:32.844044 1160 rpcz_store.cc:276] 0625 07:52:30.868425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.868536 (+ 111us) service_pool.cc:225] Handling call
0625 07:52:32.843907 (+1975371us) ts_tablet_manager.cc:1145] Deleting tablet bedc38018eb6412b9e6ca3f0ee34cf87
0625 07:52:32.843915 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.843973 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1975172}
W20250625 07:52:32.844158 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b782414a682246feb2811b4e3bf8b5a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b782414a682246feb2811b4e3bf8b5a2 already in progress: creating tablet
I20250625 07:52:32.844293 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1074) took 1966 ms. Trace:
I20250625 07:52:32.844354 1160 rpcz_store.cc:276] 0625 07:52:30.878287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:30.878400 (+ 113us) service_pool.cc:225] Handling call
0625 07:52:32.844215 (+1965815us) ts_tablet_manager.cc:1145] Deleting tablet 6881336f1ff44e3cb7d1bc8937e503a4
0625 07:52:32.844222 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.844281 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965638}
W20250625 07:52:32.844511 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bedc38018eb6412b9e6ca3f0ee34cf87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bedc38018eb6412b9e6ca3f0ee34cf87 already in progress: creating tablet
I20250625 07:52:32.844574 1160 tablet_replica.cc:331] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:32.844751 1160 raft_consensus.cc:2241] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.844903 1160 raft_consensus.cc:2270] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:32.844954 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 6881336f1ff44e3cb7d1bc8937e503a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6881336f1ff44e3cb7d1bc8937e503a4 already in progress: creating tablet
I20250625 07:52:32.845330 1160 ts_tablet_manager.cc:1905] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.845333 2162 raft_consensus.cc:513] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.845944 2162 leader_election.cc:290] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:32.846798 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:32.846940 914 raft_consensus.cc:3058] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.849587 914 raft_consensus.cc:2466] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:32.850050 713 leader_election.cc:304] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:32.850250 1162 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.850409 2162 raft_consensus.cc:2802] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:32.850556 2162 raft_consensus.cc:695] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:32.850392 1162 ts_tablet_manager.cc:1397] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20250625 07:52:32.850821 2162 consensus_queue.cc:237] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.851715 1162 raft_consensus.cc:357] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.851945 1162 raft_consensus.cc:383] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.852053 1162 raft_consensus.cc:738] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.852295 1162 consensus_queue.cc:260] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.857154 1162 ts_tablet_manager.cc:1428] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:32.857532 1162 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:32.859652 1162 tablet_bootstrap.cc:654] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.839427 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.860817 1161 log.cc:1199] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/445bfed532004108a12c8fd78370c1cb
I20250625 07:52:32.846567 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28e490ba5d08498f921b50f057b88369" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:32.861227 1161 ts_tablet_manager.cc:1939] T 445bfed532004108a12c8fd78370c1cb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.861826 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 445bfed532004108a12c8fd78370c1cb (table default.Table_6960d95c6d2f46529495f6a7be9a0d43 [id=a7c406dd64d844b7ba7d94889bb6fc10]) successfully deleted
W20250625 07:52:32.861191 713 leader_election.cc:343] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.862263 1161 tablet_replica.cc:331] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:32.862381 713 leader_election.cc:343] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.862460 2178 raft_consensus.cc:491] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.862582 2178 raft_consensus.cc:513] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.863184 2178 leader_election.cc:290] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:32.862593 1161 raft_consensus.cc:2241] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.863746 1161 raft_consensus.cc:2270] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.864286 1161 ts_tablet_manager.cc:1905] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:32.869645 986 leader_election.cc:343] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:32.875120 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_97b7cf4f260c44ac87961d4a83a7d50a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.875661 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97b7cf4f260c44ac87961d4a83a7d50a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.878263 1161 ts_tablet_manager.cc:1918] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.878561 1161 log.cc:1199] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a5a3751333bc48fc8a59b0e7b67a9935
I20250625 07:52:32.878938 1161 ts_tablet_manager.cc:1939] T a5a3751333bc48fc8a59b0e7b67a9935 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.879433 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a5a3751333bc48fc8a59b0e7b67a9935 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
I20250625 07:52:32.879593 1161 tablet_replica.cc:331] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.879828 1161 raft_consensus.cc:2241] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.879989 1161 raft_consensus.cc:2270] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.880437 1161 ts_tablet_manager.cc:1905] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:32.888260 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7c6d068d2324ea6a07bb9be74578870 consensus state has no leader
I20250625 07:52:32.889561 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_84bea214164245c8a5debd8bd9d38fb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.889653 985 leader_election.cc:343] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:32.889817 985 leader_election.cc:304] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:32.890266 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84bea214164245c8a5debd8bd9d38fb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.890496 2178 raft_consensus.cc:2747] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.894903 2178 raft_consensus.cc:491] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.895026 2178 raft_consensus.cc:513] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.895576 2178 leader_election.cc:290] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:32.896592 986 leader_election.cc:343] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a33b7cab19244ec08d3821b6cc4c072d
W20250625 07:52:32.897136 985 leader_election.cc:343] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:32.897262 985 leader_election.cc:304] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:32.897732 2178 raft_consensus.cc:2747] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:32.898684 1046 raft_consensus.cc:3058] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.898993 1158 tablet_bootstrap.cc:492] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.899098 1158 ts_tablet_manager.cc:1397] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.217s user 0.056s sys 0.087s
I20250625 07:52:32.899749 1158 raft_consensus.cc:357] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.899888 1158 raft_consensus.cc:383] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.899932 1158 raft_consensus.cc:738] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.900101 1158 consensus_queue.cc:260] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.901468 1158 ts_tablet_manager.cc:1428] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:32.901711 1158 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.903987 1158 tablet_bootstrap.cc:654] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.904776 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5d5d75af69a841d5b3961973eb62dbab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.905282 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d5d75af69a841d5b3961973eb62dbab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.912987 1161 ts_tablet_manager.cc:1918] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.913149 1161 log.cc:1199] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9418b81f010d4f369d3409e9364bc5fa
W20250625 07:52:32.917028 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a13381cae324e9494a2497d0190a868 consensus state has no leader
I20250625 07:52:32.917610 1161 ts_tablet_manager.cc:1939] T 9418b81f010d4f369d3409e9364bc5fa P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.918147 1161 tablet_replica.cc:331] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.918341 1161 raft_consensus.cc:2241] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.918329 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9418b81f010d4f369d3409e9364bc5fa (table default.Table_43f3b349ae154f228f90a77e3c33c46d [id=ccb79bd01c7746cf81b64528f5a429cd]) successfully deleted
I20250625 07:52:32.918509 1161 raft_consensus.cc:2270] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.918900 1048 raft_consensus.cc:3058] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.919036 1161 ts_tablet_manager.cc:1905] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.922582 1158 tablet_bootstrap.cc:492] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:32.922729 1158 ts_tablet_manager.cc:1397] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20250625 07:52:32.923656 1158 raft_consensus.cc:357] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.923871 1158 raft_consensus.cc:383] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.923929 1158 raft_consensus.cc:738] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.924149 1158 consensus_queue.cc:260] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.924911 1158 ts_tablet_manager.cc:1428] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250625 07:52:32.925266 1158 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:32.927282 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4208ed1cf6fc494f86365f0b22d2e390" } modify_external_catalogs: true
I20250625 07:52:32.927429 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4208ed1cf6fc494f86365f0b22d2e390" } modify_external_catalogs: true
I20250625 07:52:32.928174 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 450727d8f33e4eb489969b7974cdeeb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.930488 1158 tablet_bootstrap.cc:654] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:32.937414 1163 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:32.937568 1163 ts_tablet_manager.cc:1397] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.004s
I20250625 07:52:32.938340 1163 raft_consensus.cc:357] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.938513 1163 raft_consensus.cc:383] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.938581 1163 raft_consensus.cc:738] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.938807 1163 consensus_queue.cc:260] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:32.939286 1163 ts_tablet_manager.cc:1428] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:32.939554 1163 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:32.941869 1161 ts_tablet_manager.cc:1918] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.942297 1161 log.cc:1199] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/960ded22e2eb42d9a3494c020a4ea354
I20250625 07:52:32.942855 1161 ts_tablet_manager.cc:1939] T 960ded22e2eb42d9a3494c020a4ea354 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.944056 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 960ded22e2eb42d9a3494c020a4ea354 (table default.Table_786d6b63b6904ffe87ce1f3eb9769954 [id=e8fb18d5b2ae454aa4f4a65fab67c467]) successfully deleted
I20250625 07:52:32.945034 1161 tablet_replica.cc:331] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.945325 1161 raft_consensus.cc:2241] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.945748 1161 raft_consensus.cc:2270] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.947132 1161 ts_tablet_manager.cc:1905] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.953436 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_346480a0ddc54315b4df65b09d71294f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:32.953996 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_346480a0ddc54315b4df65b09d71294f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:32.968400 596 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet cb23f900faf6421fb6e72f2b5202d0fd (table default.Table_fffd92726b724fd09763acbb604f0941 [id=38f8a949076e4a64acdda251b315ac87]). Expected version 1 got 0
I20250625 07:52:32.969806 596 master_service.cc:948] ReplaceTablet: received request to replace tablet f35ac535578d4681ac998a530b1f1d4f from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:32.969903 1161 ts_tablet_manager.cc:1918] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:32.970007 1161 log.cc:1199] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/250175f54ba94a9fbc2455d9cf27fb2e
I20250625 07:52:32.970405 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet a630f8804bbf41de86b08cde419643ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:32.973273 1161 ts_tablet_manager.cc:1939] T 250175f54ba94a9fbc2455d9cf27fb2e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:32.976471 914 raft_consensus.cc:1273] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:32.978549 2162 consensus_queue.cc:1035] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:32.978897 2162 consensus_queue.cc:1035] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:32.983589 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 250175f54ba94a9fbc2455d9cf27fb2e (table default.Table_e0bed2d3626248cfa65f9de3024542fe [id=02b139db42304a61aecb74192e1722e6]) successfully deleted
W20250625 07:52:32.986763 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 51c584e6bffb4f0a8020dadf78d718a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c584e6bffb4f0a8020dadf78d718a7 already in progress: creating tablet
W20250625 07:52:32.988205 713 consensus_peers.cc:487] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:32.990142 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5099f01a4a124d23a7c32accbd576e54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5099f01a4a124d23a7c32accbd576e54 already in progress: creating tablet
W20250625 07:52:32.990666 713 consensus_peers.cc:487] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:32.991288 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3cdbdd1e3b734463ab452bafecb75822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdbdd1e3b734463ab452bafecb75822 already in progress: creating tablet
I20250625 07:52:32.991482 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09566a49416a48758a3abe2ff7a69087" } modify_external_catalogs: true
I20250625 07:52:32.991796 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_09566a49416a48758a3abe2ff7a69087" } modify_external_catalogs: true
W20250625 07:52:32.992311 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bd9a6cded85c4d6296f8468644de2016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd9a6cded85c4d6296f8468644de2016 already in progress: creating tablet
I20250625 07:52:32.992484 2045 raft_consensus.cc:491] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:32.992741 2045 raft_consensus.cc:513] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:32.993085 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet af8c114c1c464bcc95dbd287c88976d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af8c114c1c464bcc95dbd287c88976d6 already in progress: creating tablet
I20250625 07:52:32.993317 2045 leader_election.cc:290] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:32.993799 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:32.994207 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:32.994257 713 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:32.994415 913 raft_consensus.cc:2466] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:32.995007 713 leader_election.cc:304] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:32.995247 2045 raft_consensus.cc:2802] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:32.995325 2045 raft_consensus.cc:491] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:32.995410 2045 raft_consensus.cc:3058] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:32.995523 1161 tablet_replica.cc:331] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:32.995710 1161 raft_consensus.cc:2241] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:32.995869 1161 raft_consensus.cc:2270] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:32.996053 1162 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:32.996166 1162 ts_tablet_manager.cc:1397] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.007s
I20250625 07:52:32.996371 1161 ts_tablet_manager.cc:1905] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:32.996941 1162 raft_consensus.cc:357] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.997098 1162 raft_consensus.cc:383] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:32.997143 1162 raft_consensus.cc:738] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:32.998318 2045 raft_consensus.cc:513] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:32.948356 2218 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:32.998948 2045 leader_election.cc:290] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.000077 2171 tablet.cc:1722] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:33.005259 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_49db16d175f84573b2f9d34ce29ac535" } modify_external_catalogs: true
I20250625 07:52:33.005429 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_49db16d175f84573b2f9d34ce29ac535" } modify_external_catalogs: true
I20250625 07:52:33.005558 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.005759 914 raft_consensus.cc:3058] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.007431 2131 tablet.cc:1722] T 4b4627f4da6b4542aff1e44157087b14 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:33.008982 914 raft_consensus.cc:2466] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:33.009135 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "086c9321a70e44b5bb3ea419bb466fbe" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:33.009433 713 leader_election.cc:304] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.009724 2045 raft_consensus.cc:2802] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.009959 2045 raft_consensus.cc:695] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
W20250625 07:52:33.010033 713 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.010205 2045 consensus_queue.cc:237] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.010838 1162 consensus_queue.cc:260] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.011766 1162 ts_tablet_manager.cc:1428] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250625 07:52:33.012533 1158 tablet_bootstrap.cc:492] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.012745 1158 ts_tablet_manager.cc:1397] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.088s user 0.015s sys 0.000s
I20250625 07:52:33.013486 1158 raft_consensus.cc:357] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.013711 1158 raft_consensus.cc:383] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.013800 1158 raft_consensus.cc:738] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.014058 1158 consensus_queue.cc:260] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.019476 1158 ts_tablet_manager.cc:1428] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250625 07:52:33.019614 1160 ts_tablet_manager.cc:1918] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.019745 1160 log.cc:1199] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c0cc3ba09d5e4c3d88e74fffc97eea8b
I20250625 07:52:33.020051 1160 ts_tablet_manager.cc:1939] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.020818 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1094) took 1923 ms. Trace:
I20250625 07:52:33.020900 1160 rpcz_store.cc:276] 0625 07:52:31.096895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.097079 (+ 184us) service_pool.cc:225] Handling call
0625 07:52:32.844537 (+1747458us) ts_tablet_manager.cc:1145] Deleting tablet c0cc3ba09d5e4c3d88e74fffc97eea8b
0625 07:52:32.844545 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:32.848544 (+ 3999us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:32.946283 (+ 97739us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.020797 (+ 74514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1747261}
I20250625 07:52:33.021289 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1114) took 1464 ms. Trace:
I20250625 07:52:33.021343 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (table default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]) successfully deleted
I20250625 07:52:33.021364 1160 rpcz_store.cc:276] 0625 07:52:31.556397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.557172 (+ 775us) service_pool.cc:225] Handling call
0625 07:52:33.021166 (+1463994us) ts_tablet_manager.cc:1145] Deleting tablet 308b387d366c4d7884898937a9fc7690
0625 07:52:33.021175 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.021273 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463841}
W20250625 07:52:33.021662 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 308b387d366c4d7884898937a9fc7690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 308b387d366c4d7884898937a9fc7690 already in progress: creating tablet
I20250625 07:52:33.021801 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1113) took 1465 ms. Trace:
I20250625 07:52:33.021878 1160 rpcz_store.cc:276] 0625 07:52:31.556292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.556357 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:33.021689 (+1465332us) ts_tablet_manager.cc:1145] Deleting tablet 08cd482ceec244c2bed2b116d5c14d89
0625 07:52:33.021697 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.021766 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464198}
W20250625 07:52:33.022156 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 08cd482ceec244c2bed2b116d5c14d89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08cd482ceec244c2bed2b116d5c14d89 already in progress: creating tablet
I20250625 07:52:33.022320 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1112) took 1466 ms. Trace:
I20250625 07:52:33.022401 1160 rpcz_store.cc:276] 0625 07:52:31.556184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.557380 (+ 1196us) service_pool.cc:225] Handling call
0625 07:52:33.022176 (+1464796us) ts_tablet_manager.cc:1145] Deleting tablet 7c8abe36eab5498eb9bc3a0355101dd6
0625 07:52:33.022185 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.022304 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462644}
I20250625 07:52:33.022599 1158 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:33.022701 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7c8abe36eab5498eb9bc3a0355101dd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c8abe36eab5498eb9bc3a0355101dd6 already in progress: creating tablet
I20250625 07:52:33.023098 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:33.025656 1162 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.025770 1158 tablet_bootstrap.cc:654] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.027329 1162 tablet_bootstrap.cc:654] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.029013 1161 ts_tablet_manager.cc:1918] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.029158 1161 log.cc:1199] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2db9e8d0fefa47bfaeee462dba096b49
I20250625 07:52:33.029520 1161 ts_tablet_manager.cc:1939] T 2db9e8d0fefa47bfaeee462dba096b49 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.030234 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2db9e8d0fefa47bfaeee462dba096b49 (table default.Table_e3f2c828709a47329e6025da1ceb39a4 [id=56d4d8549fbb41e69ee20ab85a21cf8a]) successfully deleted
I20250625 07:52:33.033591 2171 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.036964 1163 tablet_bootstrap.cc:654] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.044019 1850 raft_consensus.cc:491] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.044160 1850 raft_consensus.cc:513] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.044719 1850 leader_election.cc:290] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:33.047600 713 leader_election.cc:343] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: bbe5c37fc9bd49c685de8f469a7af0c6
I20250625 07:52:33.048621 1161 tablet_replica.cc:331] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.048940 1161 raft_consensus.cc:2241] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.049095 1161 raft_consensus.cc:2270] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.049595 1161 ts_tablet_manager.cc:1905] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.049954 1850 raft_consensus.cc:491] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.050117 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe5c37fc9bd49c685de8f469a7af0c6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:33.050081 1850 raft_consensus.cc:513] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.050696 1850 leader_election.cc:290] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:33.050904 713 leader_election.cc:343] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.051038 713 leader_election.cc:304] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.051860 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.051849 597 catalog_manager.cc:5582] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.052102 914 raft_consensus.cc:2466] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:33.052203 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:33.052453 597 catalog_manager.cc:5582] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.052572 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:33.052744 713 leader_election.cc:304] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.052874 597 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]). Expected version 1 got 0
I20250625 07:52:33.052991 2162 raft_consensus.cc:2747] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.053189 2162 raft_consensus.cc:2802] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:33.053256 2162 raft_consensus.cc:491] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:33.053333 2162 raft_consensus.cc:3058] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.058143 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1fc53325704172948c557e23cf5860 (DEFAULT_TABLE table=default.Table_8f19599ce5b7472c9e5c611e68a3c16b [id=c15e0d08346a420f95a4b25c97776a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.058346 749 tablet_service.cc:1468] Processing CreateTablet for tablet 341becbce2ea45a18c9bc12d167d3072 (DEFAULT_TABLE table=default.Table_3d0f6d3d892d4afdbb88b6ed05592235 [id=1dbaf5ca08f44e1a8fac01f0636db67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.058557 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1fc53325704172948c557e23cf5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.058810 748 tablet_service.cc:1468] Processing CreateTablet for tablet b786ef91745c47638af90553a66fffdf (DEFAULT_TABLE table=default.Table_8f40c634b01a4ad19cb4a9c14cb057b7 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.059042 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341becbce2ea45a18c9bc12d167d3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.059470 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786ef91745c47638af90553a66fffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.060400 746 tablet_service.cc:1468] Processing CreateTablet for tablet 10de48baf54c40449024eadbd264ba94 (DEFAULT_TABLE table=default.Table_d92c6481d56745f9a67a5c0cdf374c9d [id=f8c88eaf700d481d8074b25161a9ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.060907 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 (DEFAULT_TABLE table=default.Table_a59e4843e6594fc8bb82fe450bb820cc [id=301e068ad3c84e86bfee7fe74371772a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.061264 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10de48baf54c40449024eadbd264ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.058560 758 tablet_service.cc:1468] Processing CreateTablet for tablet fd6c3a490883477dbafcb74d39be7345 (DEFAULT_TABLE table=default.Table_381f2b07c1fe4a47adbf6b5f45f738b5 [id=cb6c428a93c74b658c39885370fa1b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.058161 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4d9a58885c46d88327262cf73beedd (DEFAULT_TABLE table=default.Table_c6b96e7894e34dc3af94bd1f0e575a8a [id=593987d3b44445c28a57760166dbe76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063303 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 (DEFAULT_TABLE table=default.Table_a59e4843e6594fc8bb82fe450bb820cc [id=301e068ad3c84e86bfee7fe74371772a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063548 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 341becbce2ea45a18c9bc12d167d3072 (DEFAULT_TABLE table=default.Table_3d0f6d3d892d4afdbb88b6ed05592235 [id=1dbaf5ca08f44e1a8fac01f0636db67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063601 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4d9a58885c46d88327262cf73beedd (DEFAULT_TABLE table=default.Table_c6b96e7894e34dc3af94bd1f0e575a8a [id=593987d3b44445c28a57760166dbe76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063663 1012 tablet_service.cc:1468] Processing CreateTablet for tablet b786ef91745c47638af90553a66fffdf (DEFAULT_TABLE table=default.Table_8f40c634b01a4ad19cb4a9c14cb057b7 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063875 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1fc53325704172948c557e23cf5860 (DEFAULT_TABLE table=default.Table_8f19599ce5b7472c9e5c611e68a3c16b [id=c15e0d08346a420f95a4b25c97776a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.063934 1022 tablet_service.cc:1468] Processing CreateTablet for tablet fd6c3a490883477dbafcb74d39be7345 (DEFAULT_TABLE table=default.Table_381f2b07c1fe4a47adbf6b5f45f738b5 [id=cb6c428a93c74b658c39885370fa1b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.064286 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 10de48baf54c40449024eadbd264ba94 (DEFAULT_TABLE table=default.Table_d92c6481d56745f9a67a5c0cdf374c9d [id=f8c88eaf700d481d8074b25161a9ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.064720 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7d1fc53325704172948c557e23cf5860 (DEFAULT_TABLE table=default.Table_8f19599ce5b7472c9e5c611e68a3c16b [id=c15e0d08346a420f95a4b25c97776a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.064973 893 tablet_service.cc:1468] Processing CreateTablet for tablet b786ef91745c47638af90553a66fffdf (DEFAULT_TABLE table=default.Table_8f40c634b01a4ad19cb4a9c14cb057b7 [id=624ec891cba14cce84440596a5221307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.065064 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1fc53325704172948c557e23cf5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.065021 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7e4d9a58885c46d88327262cf73beedd (DEFAULT_TABLE table=default.Table_c6b96e7894e34dc3af94bd1f0e575a8a [id=593987d3b44445c28a57760166dbe76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.065245 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786ef91745c47638af90553a66fffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.065495 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 (DEFAULT_TABLE table=default.Table_a59e4843e6594fc8bb82fe450bb820cc [id=301e068ad3c84e86bfee7fe74371772a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.065492 876 tablet_service.cc:1468] Processing CreateTablet for tablet 341becbce2ea45a18c9bc12d167d3072 (DEFAULT_TABLE table=default.Table_3d0f6d3d892d4afdbb88b6ed05592235 [id=1dbaf5ca08f44e1a8fac01f0636db67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.065786 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc78b37d83b422fae1ed696d45953f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.065984 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341becbce2ea45a18c9bc12d167d3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.067127 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4d9a58885c46d88327262cf73beedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.068913 880 tablet_service.cc:1468] Processing CreateTablet for tablet fd6c3a490883477dbafcb74d39be7345 (DEFAULT_TABLE table=default.Table_381f2b07c1fe4a47adbf6b5f45f738b5 [id=cb6c428a93c74b658c39885370fa1b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.069198 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6c3a490883477dbafcb74d39be7345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.070760 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.071334 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.072926 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6c3a490883477dbafcb74d39be7345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.076360 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db110bb6cc146a393d62ddabbe1955a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:14 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.076742 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4d9a58885c46d88327262cf73beedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.077370 891 tablet_service.cc:1468] Processing CreateTablet for tablet 10de48baf54c40449024eadbd264ba94 (DEFAULT_TABLE table=default.Table_d92c6481d56745f9a67a5c0cdf374c9d [id=f8c88eaf700d481d8074b25161a9ac8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.077675 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10de48baf54c40449024eadbd264ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.080874 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc78b37d83b422fae1ed696d45953f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.080796 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 3d65025ff7c744f0aa11c714c6cddf7a (DEFAULT_TABLE table=default.Table_a9b7236721334c269477fa34a36330b8 [id=6057dfa1be244df08c97853190d4f12f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:33.082826 713 leader_election.cc:343] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.083279 2162 raft_consensus.cc:513] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.084740 2162 leader_election.cc:290] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.088747 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:33.090991 713 leader_election.cc:343] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.100787 1850 raft_consensus.cc:491] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.100941 1850 raft_consensus.cc:513] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.101434 1850 leader_election.cc:290] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.102559 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d393a85f5fde4b3fb57e5b3ec372791c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.103276 713 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d393a85f5fde4b3fb57e5b3ec372791c
W20250625 07:52:33.103511 713 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.103610 713 leader_election.cc:304] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.104185 1850 raft_consensus.cc:2747] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.105197 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1405e55703324bc7a1e26fe2c349bda7" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.105427 911 raft_consensus.cc:3058] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.108454 911 raft_consensus.cc:2466] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:33.109380 2178 raft_consensus.cc:491] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.109563 2178 raft_consensus.cc:513] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.110222 2178 leader_election.cc:290] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:33.110888 893 tablet_service.cc:1468] Processing CreateTablet for tablet 3d65025ff7c744f0aa11c714c6cddf7a (DEFAULT_TABLE table=default.Table_a9b7236721334c269477fa34a36330b8 [id=6057dfa1be244df08c97853190d4f12f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:33.111826 986 leader_election.cc:343] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:33.112344 891 tablet_service.cc:1468] Processing CreateTablet for tablet e97e1232b70d4a17829aa199f35268e8 (DEFAULT_TABLE table=default.Table_d2b9caad298b479eb23cd9548220772d [id=c938abacbfa94e0090e883088a98538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.112742 1016 tablet_service.cc:1468] Processing CreateTablet for tablet e97e1232b70d4a17829aa199f35268e8 (DEFAULT_TABLE table=default.Table_d2b9caad298b479eb23cd9548220772d [id=c938abacbfa94e0090e883088a98538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.112892 713 leader_election.cc:304] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.113562 591 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2a441f477ef7473dbd1c750b039e546a
I20250625 07:52:33.113561 1025 tablet_service.cc:1468] Processing CreateTablet for tablet c0f54f7d36a74105a2d2215682a5f3b4 (DEFAULT_TABLE table=default.Table_56030f59057b4ce39f60fb888dd0fc17 [id=311433c43b2b493e9921f83f6763eb32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.115082 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52c6475a644519828e7947e3dd46b1 (DEFAULT_TABLE table=default.Table_d6fdf8a3b2db409cb38d372aab234798 [id=bfa443add61a40ae8906ca2d8e2506bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.115216 880 tablet_service.cc:1468] Processing CreateTablet for tablet c0f54f7d36a74105a2d2215682a5f3b4 (DEFAULT_TABLE table=default.Table_56030f59057b4ce39f60fb888dd0fc17 [id=311433c43b2b493e9921f83f6763eb32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.115471 1850 raft_consensus.cc:2802] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.115636 1850 raft_consensus.cc:695] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:33.115872 1850 consensus_queue.cc:237] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.117718 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52c6475a644519828e7947e3dd46b1 (DEFAULT_TABLE table=default.Table_d6fdf8a3b2db409cb38d372aab234798 [id=bfa443add61a40ae8906ca2d8e2506bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.118116 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52c6475a644519828e7947e3dd46b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.118475 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3d65025ff7c744f0aa11c714c6cddf7a (DEFAULT_TABLE table=default.Table_a9b7236721334c269477fa34a36330b8 [id=6057dfa1be244df08c97853190d4f12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.119994 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d65025ff7c744f0aa11c714c6cddf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.120504 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4a4b98165527418ea3ccc846b86e4d15" } new_table_name: "default.Table_32aef73480b241f1abf94bdee1fc6fc3" modify_external_catalogs: true
I20250625 07:52:33.121395 758 tablet_service.cc:1468] Processing CreateTablet for tablet c0f54f7d36a74105a2d2215682a5f3b4 (DEFAULT_TABLE table=default.Table_56030f59057b4ce39f60fb888dd0fc17 [id=311433c43b2b493e9921f83f6763eb32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.121721 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f54f7d36a74105a2d2215682a5f3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.123711 2162 consensus_queue.cc:1035] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:33.126997 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 33441054063e47deaba2ad04ae6ae764
I20250625 07:52:33.127730 752 tablet_service.cc:1468] Processing CreateTablet for tablet e97e1232b70d4a17829aa199f35268e8 (DEFAULT_TABLE table=default.Table_d2b9caad298b479eb23cd9548220772d [id=c938abacbfa94e0090e883088a98538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.128057 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97e1232b70d4a17829aa199f35268e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.128665 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 33441054063e47deaba2ad04ae6ae764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.130633 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 33441054063e47deaba2ad04ae6ae764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.115229 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d65025ff7c744f0aa11c714c6cddf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.133006 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1463633d3ae24de887c29fc90b7c4c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.133327 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f54f7d36a74105a2d2215682a5f3b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:33.133586 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1463633d3ae24de887c29fc90b7c4c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.136909 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97e1232b70d4a17829aa199f35268e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.139664 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1119) took 1582 ms. Trace:
I20250625 07:52:33.139753 1160 rpcz_store.cc:276] 0625 07:52:31.556930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.566647 (+ 9717us) service_pool.cc:225] Handling call
0625 07:52:33.131403 (+1564756us) ts_tablet_manager.cc:1145] Deleting tablet 484a49fab6134d059c64f5a3f2b29e03
0625 07:52:33.139515 (+ 8112us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.139643 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564578}
W20250625 07:52:33.140197 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 484a49fab6134d059c64f5a3f2b29e03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 484a49fab6134d059c64f5a3f2b29e03 already in progress: creating tablet
I20250625 07:52:33.140597 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1121) took 1571 ms. Trace:
I20250625 07:52:33.140678 1160 rpcz_store.cc:276] 0625 07:52:31.568994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.592211 (+ 23217us) service_pool.cc:225] Handling call
0625 07:52:33.140489 (+1548278us) ts_tablet_manager.cc:1145] Deleting tablet 7be8ab5140aa4d48be978519cb99ed31
0625 07:52:33.140498 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.140582 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548081}
W20250625 07:52:33.141176 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7be8ab5140aa4d48be978519cb99ed31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7be8ab5140aa4d48be978519cb99ed31 already in progress: creating tablet
I20250625 07:52:33.141566 1160 tablet_replica.cc:331] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.141739 1160 raft_consensus.cc:2241] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.141885 1160 raft_consensus.cc:2270] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.142382 1160 ts_tablet_manager.cc:1905] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.145004 885 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52c6475a644519828e7947e3dd46b1 (DEFAULT_TABLE table=default.Table_d6fdf8a3b2db409cb38d372aab234798 [id=bfa443add61a40ae8906ca2d8e2506bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.145066 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2 (DEFAULT_TABLE table=default.Table_3a28b5bd95a545b2bde25fce57861fc2 [id=1a74653c96604a7796d217194f585a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.146046 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 33441054063e47deaba2ad04ae6ae764 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.146754 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52c6475a644519828e7947e3dd46b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.147181 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2 (DEFAULT_TABLE table=default.Table_3a28b5bd95a545b2bde25fce57861fc2 [id=1a74653c96604a7796d217194f585a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.147634 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.148113 1158 tablet_bootstrap.cc:492] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.148279 1158 ts_tablet_manager.cc:1397] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.001s
I20250625 07:52:33.148584 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc78b37d83b422fae1ed696d45953f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.149121 1158 raft_consensus.cc:357] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.149330 1158 raft_consensus.cc:383] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.149420 1158 raft_consensus.cc:738] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.149595 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 74fa6e5ac74444729645a27fc2945bc6
I20250625 07:52:33.149632 1158 consensus_queue.cc:260] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.150144 1158 ts_tablet_manager.cc:1428] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:33.150441 1158 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.152070 1158 tablet_bootstrap.cc:654] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.152731 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a1b9044e71d54104980129f70f2a399a (DEFAULT_TABLE table=default.Table_40f5f254457348eeb98731f1bab610f7 [id=b4504fff95fa46d5a02f1fe3cf6b3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.152931 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 341becbce2ea45a18c9bc12d167d3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.153688 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b9044e71d54104980129f70f2a399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.156980 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e4d9a58885c46d88327262cf73beedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.161131 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b786ef91745c47638af90553a66fffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.161334 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1fc53325704172948c557e23cf5860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.161501 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10de48baf54c40449024eadbd264ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.164959 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d65025ff7c744f0aa11c714c6cddf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.165185 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.165549 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e97e1232b70d4a17829aa199f35268e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.166209 2045 raft_consensus.cc:491] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.166326 2045 raft_consensus.cc:513] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.166862 2045 leader_election.cc:290] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.167526 2218 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.168107 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6c3a490883477dbafcb74d39be7345. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.168888 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f54f7d36a74105a2d2215682a5f3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.171947 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52c6475a644519828e7947e3dd46b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.172945 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.175560 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4167282133f94bfb9870bd08fc6594b4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.160542 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 33441054063e47deaba2ad04ae6ae764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33441054063e47deaba2ad04ae6ae764 already in progress: creating tablet
W20250625 07:52:33.179592 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 74fa6e5ac74444729645a27fc2945bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fa6e5ac74444729645a27fc2945bc6 already in progress: creating tablet
I20250625 07:52:33.183970 749 tablet_service.cc:1468] Processing CreateTablet for tablet a1b9044e71d54104980129f70f2a399a (DEFAULT_TABLE table=default.Table_40f5f254457348eeb98731f1bab610f7 [id=b4504fff95fa46d5a02f1fe3cf6b3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.184338 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b9044e71d54104980129f70f2a399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.186798 2131 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.191134 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2 (DEFAULT_TABLE table=default.Table_3a28b5bd95a545b2bde25fce57861fc2 [id=1a74653c96604a7796d217194f585a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.193734 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c5934d5265ce47728b8a5007d7b1ae14" } new_table_name: "default.Table_6caaf2d07d6b4d1585502707cea12756" modify_external_catalogs: true
I20250625 07:52:33.195019 2112 raft_consensus.cc:491] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.195333 2112 raft_consensus.cc:513] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.195901 2112 leader_election.cc:290] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.196236 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07a07af928ff4d3d894b074ef0cf44eb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:33.196722 851 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.197209 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07a07af928ff4d3d894b074ef0cf44eb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.198159 713 leader_election.cc:343] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:33.199362 713 leader_election.cc:343] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:33.199398 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b8b5bb3fe3ef4bfdba93b862784a9f61
I20250625 07:52:33.199508 713 leader_election.cc:304] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.199782 1850 raft_consensus.cc:2747] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.199879 595 catalog_manager.cc:6652] ReplaceTablet: tablet b8b5bb3fe3ef4bfdba93b862784a9f61 deleted and replaced by tablet bc387934e5204a3e9ce6f13d1dc3f777
W20250625 07:52:33.202149 850 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.202292 850 leader_election.cc:304] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.202533 2112 raft_consensus.cc:2747] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.202961 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.204072 1161 ts_tablet_manager.cc:1918] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.206796 1163 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.206936 1163 ts_tablet_manager.cc:1397] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.267s user 0.004s sys 0.000s
I20250625 07:52:33.207681 1163 raft_consensus.cc:357] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.207828 1163 raft_consensus.cc:383] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.207876 1163 raft_consensus.cc:738] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.208046 1163 consensus_queue.cc:260] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.208500 1163 ts_tablet_manager.cc:1428] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250625 07:52:33.208554 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 169794us
I20250625 07:52:33.210619 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3aebb6670cd64a50af22d28be0a65e47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.211191 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aebb6670cd64a50af22d28be0a65e47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.214592 891 tablet_service.cc:1515] Processing DeleteTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 with delete_type TABLET_DATA_DELETED (replaced by tablet bc387934e5204a3e9ce6f13d1dc3f777) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.217597 880 tablet_service.cc:1468] Processing CreateTablet for tablet a1b9044e71d54104980129f70f2a399a (DEFAULT_TABLE table=default.Table_40f5f254457348eeb98731f1bab610f7 [id=b4504fff95fa46d5a02f1fe3cf6b3839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.219002 1161 log.cc:1199] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3ca0144e3c5142ac9fcbd927b7585a3c
I20250625 07:52:33.219449 1161 ts_tablet_manager.cc:1939] T 3ca0144e3c5142ac9fcbd927b7585a3c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.210908 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 with delete_type TABLET_DATA_DELETED (replaced by tablet bc387934e5204a3e9ce6f13d1dc3f777) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.211616 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bed3f04c60249aaa389c26788974f58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:33.219496 985 leader_election.cc:343] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:33.219985 985 leader_election.cc:304] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.220196 2178 raft_consensus.cc:2747] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.211755 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 74fa6e5ac74444729645a27fc2945bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.222111 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3ca0144e3c5142ac9fcbd927b7585a3c (table default.Table_dfc00fb110744489ac79c5a1e56bf3c5 [id=b89fd2b709f943159ddbbc27846046b1]) successfully deleted
I20250625 07:52:33.222225 1161 tablet_replica.cc:331] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.222517 1161 raft_consensus.cc:2241] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.222775 1161 raft_consensus.cc:2270] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.224089 1161 ts_tablet_manager.cc:1905] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.225193 2112 raft_consensus.cc:491] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.225356 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a2481ca7e8db42a3a51270158bb2e2fa
I20250625 07:52:33.225328 2112 raft_consensus.cc:513] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.225934 2112 leader_election.cc:290] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.226955 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b7daf759e324c1db853fdbaab76c721" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.227456 850 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:33.227977 713 consensus_peers.cc:487] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:33.228243 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.232955 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.234742 1162 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.234967 1162 ts_tablet_manager.cc:1397] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.209s user 0.025s sys 0.048s
I20250625 07:52:33.235795 1162 raft_consensus.cc:357] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.236086 1162 raft_consensus.cc:383] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.236207 1162 raft_consensus.cc:738] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.236488 1162 consensus_queue.cc:260] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.236797 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet a2481ca7e8db42a3a51270158bb2e2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.236830 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b7daf759e324c1db853fdbaab76c721" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:33.237049 2045 raft_consensus.cc:491] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.237226 2045 raft_consensus.cc:513] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:33.237378 851 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.237490 851 leader_election.cc:304] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.237882 2112 raft_consensus.cc:2747] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.237903 2045 leader_election.cc:290] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.237890 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6735bc11fe9c40b78d6d3b4f2c1b9ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.239059 1162 ts_tablet_manager.cc:1428] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:33.243957 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd] alter complete (version 3)
W20250625 07:52:33.244323 713 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:33.246869 1162 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.247736 1163 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:33.248556 1162 tablet_bootstrap.cc:654] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.249230 1163 tablet_bootstrap.cc:654] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.250128 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.250386 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfacdde6a8ff4b6eac40d83c1d5b21f6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.250926 713 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.251050 713 leader_election.cc:304] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.251345 2045 raft_consensus.cc:2747] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.253892 1161 ts_tablet_manager.cc:1918] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.256089 1161 log.cc:1199] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a33b7cab19244ec08d3821b6cc4c072d
I20250625 07:52:33.256474 1161 ts_tablet_manager.cc:1939] T a33b7cab19244ec08d3821b6cc4c072d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.256843 746 tablet_service.cc:1515] Processing DeleteTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 with delete_type TABLET_DATA_DELETED (replaced by tablet bc387934e5204a3e9ce6f13d1dc3f777) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.257324 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a33b7cab19244ec08d3821b6cc4c072d (table default.Table_2f89a55bc2694564b94bb87f42761d2a [id=dff8e8b2f6da4c39b78f536bcc8467e5]) successfully deleted
I20250625 07:52:33.257647 1161 tablet_replica.cc:331] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:33.257747 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f2ee3980ed9747af87329fd3a1dfd699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ee3980ed9747af87329fd3a1dfd699 already in progress: creating tablet
I20250625 07:52:33.258095 1161 raft_consensus.cc:2241] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.258991 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b9044e71d54104980129f70f2a399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.260632 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cc8b946dd3564d4b95fa7174c5c9bd96" } new_table_name: "default.Table_afa358a8ea9a4e999a4160be5d7f51ed" modify_external_catalogs: true
I20250625 07:52:33.266920 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f35ac535578d4681ac998a530b1f1d4f
I20250625 07:52:33.267300 596 catalog_manager.cc:6652] ReplaceTablet: tablet f35ac535578d4681ac998a530b1f1d4f deleted and replaced by tablet a17495ae62b340258225c78df131b9c3
I20250625 07:52:33.258821 1158 tablet_bootstrap.cc:492] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.268821 1158 ts_tablet_manager.cc:1397] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.000s
I20250625 07:52:33.268813 1161 raft_consensus.cc:2270] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.271077 746 tablet_service.cc:1515] Processing DeleteTablet for tablet f35ac535578d4681ac998a530b1f1d4f with delete_type TABLET_DATA_DELETED (replaced by tablet a17495ae62b340258225c78df131b9c3) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.271174 880 tablet_service.cc:1515] Processing DeleteTablet for tablet f35ac535578d4681ac998a530b1f1d4f with delete_type TABLET_DATA_DELETED (replaced by tablet a17495ae62b340258225c78df131b9c3) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.269376 1161 ts_tablet_manager.cc:1905] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.269604 1158 raft_consensus.cc:357] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.271452 1158 raft_consensus.cc:383] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.271526 1158 raft_consensus.cc:738] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.271698 1158 consensus_queue.cc:260] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.272114 1158 ts_tablet_manager.cc:1428] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:33.272392 1158 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.273738 1158 tablet_bootstrap.cc:654] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.270938 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet f35ac535578d4681ac998a530b1f1d4f with delete_type TABLET_DATA_DELETED (replaced by tablet a17495ae62b340258225c78df131b9c3) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.282625 2045 consensus_queue.cc:1035] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:52:33.282701 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a2481ca7e8db42a3a51270158bb2e2fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2481ca7e8db42a3a51270158bb2e2fa already in progress: creating tablet
I20250625 07:52:33.292171 1158 tablet_bootstrap.cc:492] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.292356 1158 ts_tablet_manager.cc:1397] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20250625 07:52:33.293030 1161 ts_tablet_manager.cc:1918] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.293190 1161 log.cc:1199] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/29f2997d1d714100950383e6af04f315
I20250625 07:52:33.293172 1158 raft_consensus.cc:357] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.295430 1158 raft_consensus.cc:383] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.295542 1158 raft_consensus.cc:738] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.295742 1161 ts_tablet_manager.cc:1939] T 29f2997d1d714100950383e6af04f315 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.295850 1158 consensus_queue.cc:260] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.300580 1158 ts_tablet_manager.cc:1428] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 07:52:33.299110 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 29f2997d1d714100950383e6af04f315 (table default.Table_c24058d938ab46a3a19827cf537af456 [id=bc67649b4b824be7b2563eb5f41247cd]) successfully deleted
W20250625 07:52:33.302300 597 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:33657 (request call id 221) took 821 ms (client timeout 1000 ms). Trace:
W20250625 07:52:33.304392 597 rpcz_store.cc:269] 0625 07:52:32.481079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.494276 (+ 13197us) service_pool.cc:225] Handling call
0625 07:52:32.505290 (+ 11014us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 00007feeb473b14d 00007feeb473b243 00007feeb13fe341 00007feeb8f7fd8d 00007feeb91035f8 00007feeaff4cb36 00007feeaff4c865 00007feeac73ec7e 00007feeac73db5e 00007feeac743d95 00007feeac7446bf
0625 07:52:33.302239 (+796949us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:33.279640 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:33.279742 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:33.279748 (+ 6us) write_op.cc:435] Acquired schema lock
0625 07:52:33.279750 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:33.279891 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:33.279907 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:33.279911 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0625 07:52:33.279997 (+ 86us) tablet.cc:699] Row locks acquired
0625 07:52:33.280000 (+ 3us) write_op.cc:260] PREPARE: finished
0625 07:52:33.280029 (+ 29us) write_op.cc:270] Start()
0625 07:52:33.280054 (+ 25us) write_op.cc:276] Timestamp: P: 1750837953280012 usec, L: 0
0625 07:52:33.280058 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:33.280231 (+ 173us) log.cc:844] Serialized 901 byte log entry
0625 07:52:33.300245 (+ 20014us) op_driver.cc:464] REPLICATION: finished
0625 07:52:33.300416 (+ 171us) write_op.cc:301] APPLY: starting
0625 07:52:33.300431 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:33.300497 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:33.300522 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:33.300590 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:33.300628 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 07:52:33.300634 (+ 6us) write_op.cc:312] APPLY: finished
0625 07:52:33.300730 (+ 96us) log.cc:844] Serialized 29 byte log entry
0625 07:52:33.300795 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:33.300825 (+ 30us) write_op.cc:454] Released schema lock
0625 07:52:33.300832 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":24002816,"child_traces":[["op",{"apply.queue_time_us":125,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":477,"num_ops":2,"prepare.queue_time_us":226118,"prepare.run_cpu_time_us":762,"prepare.run_wall_time_us":25742,"raft.queue_time_us":43,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":20157}]]}
I20250625 07:52:33.306536 1158 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.306092 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f3b863e17f8942c58fe854c0dab420be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.307407 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b863e17f8942c58fe854c0dab420be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.308261 1158 tablet_bootstrap.cc:654] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.309739 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 68f903943e9a4151b6fa1e71d1c05473
I20250625 07:52:33.310081 756 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.310340 1166 tablet_replica.cc:331] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:33.310564 1166 raft_consensus.cc:2241] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.310772 1166 raft_consensus.cc:2270] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.311326 1166 ts_tablet_manager.cc:1905] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.311569 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:33.312932 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b5bb3fe3ef4bfdba93b862784a9f61 already in progress: creating tablet
W20250625 07:52:33.313154 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f35ac535578d4681ac998a530b1f1d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f35ac535578d4681ac998a530b1f1d4f already in progress: creating tablet
I20250625 07:52:33.313298 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.314587 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b1873a0532e24b77925f3c5b029dcda3" } new_table_name: "default.Table_ee665ce3235a4226b7bca8e30b6ef86f" modify_external_catalogs: true
I20250625 07:52:33.317478 1162 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.317620 1162 ts_tablet_manager.cc:1397] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.000s
I20250625 07:52:33.318385 1162 raft_consensus.cc:357] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.318503 1166 ts_tablet_manager.cc:1918] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.319562 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f903943e9a4151b6fa1e71d1c05473 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:32 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.318557 1162 raft_consensus.cc:383] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.319185 756 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.319993 1162 raft_consensus.cc:738] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.320329 1162 consensus_queue.cc:260] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.320626 1166 log.cc:1199] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:33.321079 1166 ts_tablet_manager.cc:1939] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:33.323658 1163 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.321976 1162 ts_tablet_manager.cc:1428] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:33.323799 1163 ts_tablet_manager.cc:1397] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.000s
I20250625 07:52:33.324060 1162 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.324519 1163 raft_consensus.cc:357] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.324676 1163 raft_consensus.cc:383] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.324776 1163 raft_consensus.cc:738] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.325076 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (table default.Table_e57f684466224dd5a4a8ff79408e1a4d [id=3dcf0706033c48ceaa1fdd5763284bf0]) successfully deleted
I20250625 07:52:33.325192 1163 consensus_queue.cc:260] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.325688 1163 ts_tablet_manager.cc:1428] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 07:52:33.325636 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8a899790713d4646a850971bc3aebd9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a899790713d4646a850971bc3aebd9f already in progress: creating tablet
I20250625 07:52:33.325928 1163 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:33.326242 713 consensus_peers.cc:487] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:33.326478 1161 tablet_replica.cc:331] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.326689 1161 raft_consensus.cc:2241] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.327030 1161 raft_consensus.cc:2270] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.327399 1163 tablet_bootstrap.cc:654] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:33.327930 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 68f903943e9a4151b6fa1e71d1c05473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f903943e9a4151b6fa1e71d1c05473 already in progress: creating tablet
I20250625 07:52:33.328279 1166 tablet_replica.cc:331] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:33.328486 1166 raft_consensus.cc:2241] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.328750 1161 ts_tablet_manager.cc:1905] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.329033 1166 raft_consensus.cc:2270] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.330240 1850 raft_consensus.cc:491] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.334403 1850 raft_consensus.cc:513] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.334944 1166 ts_tablet_manager.cc:1905] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.335036 1850 leader_election.cc:290] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.335806 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:33.336231 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.336391 911 raft_consensus.cc:2466] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:33.336445 713 leader_election.cc:343] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.337010 713 leader_election.cc:304] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.337832 1850 raft_consensus.cc:2802] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:33.337932 1850 raft_consensus.cc:491] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:33.338030 1850 raft_consensus.cc:3058] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.340833 2178 raft_consensus.cc:491] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.340966 2178 raft_consensus.cc:513] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.340611 1850 raft_consensus.cc:513] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:33.341334 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cdee1535e5a4a21804e28e78aaea0eb consensus state has no leader
I20250625 07:52:33.341545 2178 leader_election.cc:290] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.341945 1850 leader_election.cc:290] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.342052 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:33.342964 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89c703dc3249408faeb522cef0248111" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.343386 985 leader_election.cc:304] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.343487 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:33.343834 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.343905 2178 raft_consensus.cc:2747] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.344007 911 raft_consensus.cc:3058] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:33.344009 713 leader_election.cc:343] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.345690 1160 ts_tablet_manager.cc:1918] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.345778 1160 log.cc:1199] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6e5a207d515449e298936d8bb532437a
I20250625 07:52:33.346040 1160 ts_tablet_manager.cc:1939] T 6e5a207d515449e298936d8bb532437a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.346588 911 raft_consensus.cc:2466] T 4a2f72b227e14e46983c906520c29fa3 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:33.347142 713 leader_election.cc:304] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.347561 1850 raft_consensus.cc:2802] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.347756 1850 raft_consensus.cc:695] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:33.348224 1850 consensus_queue.cc:237] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.350909 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_596a78c3ff9e44158b80f0e0637d57ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.351593 2112 raft_consensus.cc:491] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.351743 2112 raft_consensus.cc:513] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:33.352109 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_596a78c3ff9e44158b80f0e0637d57ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.352321 2112 leader_election.cc:290] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.353520 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c263b671be384cd097041c60306034de" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:33.354017 851 leader_election.cc:343] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.354270 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c263b671be384cd097041c60306034de" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.354696 850 leader_election.cc:343] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.354805 850 leader_election.cc:304] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.355008 2112 raft_consensus.cc:2747] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.359759 1161 ts_tablet_manager.cc:1918] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.364202 1166 ts_tablet_manager.cc:1918] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.364398 1166 log.cc:1199] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:33.364531 1161 log.cc:1199] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/494cacf2dcbb40d3b2c9f05f64809870
I20250625 07:52:33.364969 1166 ts_tablet_manager.cc:1939] T b2bb060cdd9a48688967989d7491cfc1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:33.365278 1161 ts_tablet_manager.cc:1939] T 494cacf2dcbb40d3b2c9f05f64809870 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.365903 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 494cacf2dcbb40d3b2c9f05f64809870 (table default.Table_2e804b4479be4226bc14ca030cdb1144 [id=1de2fe8568db44dabaa40fa627d650fc]) successfully deleted
I20250625 07:52:33.366776 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet b2bb060cdd9a48688967989d7491cfc1 (table default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]) successfully deleted
W20250625 07:52:33.367419 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8d34bfe377ba421daede5ae68848a2d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d34bfe377ba421daede5ae68848a2d4 already in progress: creating tablet
I20250625 07:52:33.367755 1161 tablet_replica.cc:331] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.367962 1161 raft_consensus.cc:2241] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.368140 1161 raft_consensus.cc:2270] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.368992 1161 ts_tablet_manager.cc:1905] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.373191 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f1f321543e7342108ad8b10b3c846822" } new_table_name: "default.Table_5e35537c3d4944689eda65dc794f68e6" modify_external_catalogs: true
I20250625 07:52:33.386462 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1125) took 1425 ms. Trace:
I20250625 07:52:33.386588 1160 rpcz_store.cc:276] 0625 07:52:31.960980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.961508 (+ 528us) service_pool.cc:225] Handling call
0625 07:52:33.141516 (+1180008us) ts_tablet_manager.cc:1145] Deleting tablet 6e5a207d515449e298936d8bb532437a
0625 07:52:33.141526 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.254560 (+113034us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.345586 (+ 91026us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.385927 (+ 40341us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x61800000d518. stack: 00007f8e6ebed336 00007f8e6ebed0f0 00007f8e6dbe33ad 0000000000971004 0000000000970fcd 000000000097099a 00007f8e6ebd5c3d 00007f8e6ebd6cae 00007f8e8226b80d 00007f8e821ed0f3 00007f8e7e4404e0 00007f8e7e44a5da 00007f8e7e46957a 00007f8e7e469550 00007f8e7e46951e 00007f8e7e456fd7
0625 07:52:33.386433 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4410624,"tablet-delete.queue_time_us":1179799}
I20250625 07:52:33.387055 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6e5a207d515449e298936d8bb532437a (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
I20250625 07:52:33.387501 1160 tablet_replica.cc:331] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.387745 1160 raft_consensus.cc:2241] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.387948 1160 raft_consensus.cc:2270] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.388518 1160 ts_tablet_manager.cc:1905] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.393899 1158 tablet_bootstrap.cc:492] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.394081 1158 ts_tablet_manager.cc:1397] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.088s user 0.015s sys 0.020s
I20250625 07:52:33.395115 1158 raft_consensus.cc:357] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.395349 1158 raft_consensus.cc:383] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.395402 1158 raft_consensus.cc:738] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.395645 1158 consensus_queue.cc:260] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.397231 1158 ts_tablet_manager.cc:1428] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:33.397513 1158 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.398859 1162 tablet_bootstrap.cc:654] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.399192 1158 tablet_bootstrap.cc:654] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.402431 1162 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.402639 1162 ts_tablet_manager.cc:1397] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.001s
I20250625 07:52:33.403106 1163 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.403268 1163 ts_tablet_manager.cc:1397] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.001s
I20250625 07:52:33.403757 1162 raft_consensus.cc:357] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.403993 1162 raft_consensus.cc:383] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.403909 1163 raft_consensus.cc:357] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.404095 1162 raft_consensus.cc:738] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.404152 1163 raft_consensus.cc:383] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.404230 1163 raft_consensus.cc:738] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.404458 1162 consensus_queue.cc:260] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.404541 1163 consensus_queue.cc:260] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.405022 1162 ts_tablet_manager.cc:1428] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:33.405054 1163 ts_tablet_manager.cc:1428] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:33.405370 1162 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.405718 1163 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:33.407230 1163 tablet_bootstrap.cc:654] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.407377 1162 tablet_bootstrap.cc:654] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.409209 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:33.410169 1162 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.410306 1162 ts_tablet_manager.cc:1397] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250625 07:52:33.411157 1162 raft_consensus.cc:357] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.411317 1162 raft_consensus.cc:383] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.411399 1162 raft_consensus.cc:738] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.411623 1162 consensus_queue.cc:260] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.412099 1162 ts_tablet_manager.cc:1428] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:33.412358 1162 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:33.413198 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 841d3a3c98af45339836d9849ff712fd consensus state has no leader
I20250625 07:52:33.414043 1162 tablet_bootstrap.cc:654] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.417508 1158 tablet_bootstrap.cc:492] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.417659 1158 ts_tablet_manager.cc:1397] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250625 07:52:33.418404 1158 raft_consensus.cc:357] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.418592 1158 raft_consensus.cc:383] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.418659 1158 raft_consensus.cc:738] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.418884 1158 consensus_queue.cc:260] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.419490 1158 ts_tablet_manager.cc:1428] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:33.419757 1158 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.421970 1158 tablet_bootstrap.cc:654] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.424201 2171 tablet.cc:1722] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.425083 2131 tablet.cc:1722] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.442669 2131 tablet.cc:1722] T f7f82e83ed924d2bbedd33d96642a10a P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:33.446570 2171 tablet.cc:1722] T f7f82e83ed924d2bbedd33d96642a10a P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:33.452399 2178 raft_consensus.cc:491] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.452565 2178 raft_consensus.cc:513] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.453197 2178 leader_election.cc:290] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:33.454695 1850 raft_consensus.cc:491] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.454877 1850 raft_consensus.cc:513] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.455559 1850 leader_election.cc:290] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.456096 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.456384 986 leader_election.cc:343] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: d76681f162074b67a6b886887b5a3d61
I20250625 07:52:33.456933 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.457106 911 raft_consensus.cc:2466] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:33.457568 713 leader_election.cc:304] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.457798 1850 raft_consensus.cc:2802] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:33.457880 1850 raft_consensus.cc:491] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:33.457934 1850 raft_consensus.cc:3058] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.461299 1850 raft_consensus.cc:513] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.461889 1850 leader_election.cc:290] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.462386 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:33.463186 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34a338005a7b44b9b0fba7a7096cfcb8" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.463346 911 raft_consensus.cc:3058] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.466063 911 raft_consensus.cc:2466] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:33.468555 713 leader_election.cc:304] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.468883 1850 raft_consensus.cc:2802] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.469002 1850 raft_consensus.cc:695] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:33.469228 1850 consensus_queue.cc:237] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.470903 1161 ts_tablet_manager.cc:1918] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.471004 1161 log.cc:1199] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/450727d8f33e4eb489969b7974cdeeb1
I20250625 07:52:33.471309 1161 ts_tablet_manager.cc:1939] T 450727d8f33e4eb489969b7974cdeeb1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.471691 1161 tablet_replica.cc:331] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:33.476866 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60edc8fa184542a0b54b89736831730d consensus state has no leader
I20250625 07:52:33.480147 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_53bbd615e4ad4b2a8def349b90e9cef8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.480679 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53bbd615e4ad4b2a8def349b90e9cef8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.482537 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6af0d69ef30c4a8b92133ce2a3e3bb9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.483002 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6af0d69ef30c4a8b92133ce2a3e3bb9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.484213 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d3b790bb56d44a24b68373d465cf0a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.484938 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d9b9ca707e724747aa3b189289a21de4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.485347 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9b9ca707e724747aa3b189289a21de4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:33.484652 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3b790bb56d44a24b68373d465cf0a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.488191 1158 tablet_bootstrap.cc:492] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.488332 1158 ts_tablet_manager.cc:1397] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.015s
I20250625 07:52:33.489144 1158 raft_consensus.cc:357] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.489347 1158 raft_consensus.cc:383] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.489634 1158 raft_consensus.cc:738] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.489915 1158 consensus_queue.cc:260] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.492542 2178 raft_consensus.cc:491] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.492664 2178 raft_consensus.cc:513] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.493209 2178 leader_election.cc:290] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:33.493659 1158 ts_tablet_manager.cc:1428] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:33.494019 1163 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.494146 1163 ts_tablet_manager.cc:1397] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.002s
I20250625 07:52:33.494799 1163 raft_consensus.cc:357] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.494951 1163 raft_consensus.cc:383] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.494999 1163 raft_consensus.cc:738] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.495210 1163 consensus_queue.cc:260] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.495301 1158 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.495561 1046 raft_consensus.cc:3058] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.495862 1163 ts_tablet_manager.cc:1428] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250625 07:52:33.495991 986 leader_election.cc:343] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c8c015c202544914b74f1e245b496f82
I20250625 07:52:33.496109 1163 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:33.497320 1158 tablet_bootstrap.cc:654] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.497328 1163 tablet_bootstrap.cc:654] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:33.501338 713 leader_election.cc:343] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:33.501653 713 leader_election.cc:343] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.503002 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 450727d8f33e4eb489969b7974cdeeb1 (table default.Table_5c23014bc5574cc9a64e940e87188050 [id=01cb78211cdc4a9a9b493e31f8c44a99]) successfully deleted
W20250625 07:52:33.503408 985 leader_election.cc:343] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c8c015c202544914b74f1e245b496f82
I20250625 07:52:33.503532 985 leader_election.cc:304] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.503836 2178 raft_consensus.cc:2747] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:33.503890 985 leader_election.cc:343] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d76681f162074b67a6b886887b5a3d61
I20250625 07:52:33.503964 985 leader_election.cc:304] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.504319 2178 raft_consensus.cc:2747] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.505638 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:33.515022 1162 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.515210 1162 ts_tablet_manager.cc:1397] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.013s
I20250625 07:52:33.516906 1162 raft_consensus.cc:357] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.517078 1162 raft_consensus.cc:383] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.517127 1162 raft_consensus.cc:738] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.517305 1162 consensus_queue.cc:260] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.517419 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 034903d699584feb91f35cbb2f62701a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:33.517758 1162 ts_tablet_manager.cc:1428] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:33.518074 1162 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:33.518061 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe3ccad4a5a34bfba36c5c18ec5038c0 consensus state has no leader
I20250625 07:52:33.518524 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_434924b9e2f44c0186b4b8958f24e778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.519198 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_434924b9e2f44c0186b4b8958f24e778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.519999 1162 tablet_bootstrap.cc:654] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.524957 1161 raft_consensus.cc:2241] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.525166 1161 raft_consensus.cc:2270] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.525763 1161 ts_tablet_manager.cc:1905] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.525959 1048 raft_consensus.cc:3058] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:33.536168 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d4be984846344bbb20a1076112b4330 consensus state has no leader
I20250625 07:52:33.557168 1160 ts_tablet_manager.cc:1918] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.557403 1160 log.cc:1199] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:33.557855 1160 ts_tablet_manager.cc:1939] T faf049714a7e4d7eb947c29f272bbb7c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.558822 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1124) took 1598 ms. Trace:
I20250625 07:52:33.558967 1160 rpcz_store.cc:276] 0625 07:52:31.960599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.969259 (+ 8660us) service_pool.cc:225] Handling call
0625 07:52:33.387418 (+1418159us) ts_tablet_manager.cc:1145] Deleting tablet faf049714a7e4d7eb947c29f272bbb7c
0625 07:52:33.387429 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.457903 (+ 70474us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.557020 (+ 99117us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.558800 (+ 1780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417920}
I20250625 07:52:33.559430 1160 tablet_replica.cc:331] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.559674 1160 raft_consensus.cc:2241] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.559707 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet faf049714a7e4d7eb947c29f272bbb7c (table default.Table_4c6dba196ca04a6ba70a9180b32eeec3 [id=a1e6b3ecd3bb49ef94da294bb866f824]) successfully deleted
I20250625 07:52:33.559892 1160 raft_consensus.cc:2270] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.560840 1160 ts_tablet_manager.cc:1905] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.562398 1162 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.562537 1162 ts_tablet_manager.cc:1397] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.007s
I20250625 07:52:33.563241 1162 raft_consensus.cc:357] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.563452 1162 raft_consensus.cc:383] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.563519 1162 raft_consensus.cc:738] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.563704 1162 consensus_queue.cc:260] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.568666 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_38f6c0a3b6e644eeb000b074c6393e2f" } modify_external_catalogs: true
I20250625 07:52:33.568907 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_38f6c0a3b6e644eeb000b074c6393e2f" } modify_external_catalogs: true
I20250625 07:52:33.569487 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a85de53a06a54f77a07282be4323dd74" } modify_external_catalogs: true
I20250625 07:52:33.569643 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a85de53a06a54f77a07282be4323dd74" } modify_external_catalogs: true
I20250625 07:52:33.571420 589 master_service.cc:948] ReplaceTablet: received request to replace tablet b98c1f91e4914fb29c6b413e92cb1929 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:33.571985 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57bbb43218994668b3c285c21c088ffa" } modify_external_catalogs: true
I20250625 07:52:33.572135 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57bbb43218994668b3c285c21c088ffa" } modify_external_catalogs: true
I20250625 07:52:33.573004 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_defbe4d5e7684ec5ac3c36f8d600909c" } modify_external_catalogs: true
I20250625 07:52:33.573164 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_defbe4d5e7684ec5ac3c36f8d600909c" } modify_external_catalogs: true
I20250625 07:52:33.573653 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_032dd0d1cc4f45a685a90e4fb6c3d3a5" } modify_external_catalogs: true
I20250625 07:52:33.573812 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_032dd0d1cc4f45a685a90e4fb6c3d3a5" } modify_external_catalogs: true
I20250625 07:52:33.577033 1162 ts_tablet_manager.cc:1428] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250625 07:52:33.577754 1158 tablet_bootstrap.cc:492] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.577894 1158 ts_tablet_manager.cc:1397] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.014s
I20250625 07:52:33.578563 1158 raft_consensus.cc:357] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.578776 1158 raft_consensus.cc:383] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.578831 1158 raft_consensus.cc:738] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.578998 1158 consensus_queue.cc:260] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.579509 1158 ts_tablet_manager.cc:1428] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:33.579663 1161 ts_tablet_manager.cc:1918] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.579762 1158 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.584547 1158 tablet_bootstrap.cc:654] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.585140 1162 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.586622 1162 tablet_bootstrap.cc:654] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.589946 1163 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.590073 1163 ts_tablet_manager.cc:1397] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.000s
I20250625 07:52:33.590016 2218 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:33.590963 1163 raft_consensus.cc:357] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.591118 1163 raft_consensus.cc:383] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.591161 1163 raft_consensus.cc:738] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.592433 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b975ec5e79ac42bfaa61d193c8ac170e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.591356 1163 consensus_queue.cc:260] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.593303 1163 ts_tablet_manager.cc:1428] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:33.593564 1163 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:33.598824 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b975ec5e79ac42bfaa61d193c8ac170e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.589123 2162 raft_consensus.cc:491] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.600965 2162 raft_consensus.cc:513] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.601543 2162 leader_election.cc:290] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.602052 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "547b758c2e6c4558bb232f14b37aa81a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.602557 713 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.606585 1158 tablet_bootstrap.cc:492] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.606549 2218 tablet.cc:1722] T 1228018baf3340d99b3f94b150d0d533 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:33.606709 1158 ts_tablet_manager.cc:1397] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20250625 07:52:33.579777 1161 log.cc:1199] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a630f8804bbf41de86b08cde419643ff
I20250625 07:52:33.607169 1161 ts_tablet_manager.cc:1939] T a630f8804bbf41de86b08cde419643ff P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.607607 1158 raft_consensus.cc:357] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.607769 1158 raft_consensus.cc:383] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.607836 1158 raft_consensus.cc:738] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:33.607808 713 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 547b758c2e6c4558bb232f14b37aa81a
I20250625 07:52:33.607915 713 leader_election.cc:304] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.608027 1158 consensus_queue.cc:260] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.608398 1161 tablet_replica.cc:331] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.608487 1158 ts_tablet_manager.cc:1428] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:33.608749 1158 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.610427 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a630f8804bbf41de86b08cde419643ff (table default.Table_929703cfa29e467c9090ff84a8b2cec1 [id=cd9d83bd28e5422188207277162018a1]) successfully deleted
I20250625 07:52:33.611002 2162 raft_consensus.cc:2747] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.611006 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1f1b83a6ed50499c9f7101fdf3c2ea62" } new_table_name: "default.Table_586f18dcb3f5429bbb79072497975a59" modify_external_catalogs: true
I20250625 07:52:33.611158 2162 raft_consensus.cc:491] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.611280 2162 raft_consensus.cc:513] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.611801 2162 leader_election.cc:290] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.613003 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.613171 911 raft_consensus.cc:2466] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:33.613512 1158 tablet_bootstrap.cc:654] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.613531 713 leader_election.cc:304] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.613762 1850 raft_consensus.cc:2802] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:33.613826 1850 raft_consensus.cc:491] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:33.613883 1850 raft_consensus.cc:3058] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.616272 1850 raft_consensus.cc:513] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.617117 1850 leader_election.cc:290] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.617901 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.618053 911 raft_consensus.cc:3058] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:33.608551 1161 raft_consensus.cc:2241] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.618340 1161 raft_consensus.cc:2270] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.618834 1161 ts_tablet_manager.cc:1905] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:33.620066 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 833cb1c75c5e429dac3428784007ab40 consensus state has no leader
I20250625 07:52:33.620513 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4db897967cc14088a2778777bce52c54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.620644 911 raft_consensus.cc:2466] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:33.621104 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4db897967cc14088a2778777bce52c54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.621352 713 leader_election.cc:304] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.621654 1850 raft_consensus.cc:2802] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.621768 1850 raft_consensus.cc:695] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:33.622009 1850 consensus_queue.cc:237] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.623287 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.624200 713 leader_election.cc:343] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.624639 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:33.625079 713 leader_election.cc:343] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.627736 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57825aadc5bf48329eb4259fe95fba1e" } new_table_name: "default.Table_b410796b587a4d6dba010595d955f3a9" modify_external_catalogs: true
I20250625 07:52:33.632082 1161 ts_tablet_manager.cc:1918] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.632180 1161 log.cc:1199] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7db110bb6cc146a393d62ddabbe1955a
W20250625 07:52:33.632197 584 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 99421eff752142348c8a899e7ac961b1 (table default.Table_deb382f48ba949ed979b457ddbd62fc1 [id=bf668cc9e9db4f529d2badcae5254439], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 99421eff752142348c8a899e7ac961b1 consensus state has no leader
I20250625 07:52:33.632520 1161 ts_tablet_manager.cc:1939] T 7db110bb6cc146a393d62ddabbe1955a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.633306 1162 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.633495 1162 ts_tablet_manager.cc:1397] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.002s
I20250625 07:52:33.636801 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7db110bb6cc146a393d62ddabbe1955a (table default.Table_1351bcbc744a4e77a2f86a998e3073d2 [id=cc14da16f4aa4bdcb68be381d688528b]) successfully deleted
W20250625 07:52:33.637569 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d2f251602454a1eadf4e2803e226531 consensus state has no leader
I20250625 07:52:33.638700 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea49bfdf46d04bdf936e7c4cdf8af4d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.639392 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea49bfdf46d04bdf936e7c4cdf8af4d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:33.640228 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 33441054063e47deaba2ad04ae6ae764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33441054063e47deaba2ad04ae6ae764 already in progress: creating tablet
W20250625 07:52:33.640869 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8b5bb3fe3ef4bfdba93b862784a9f61 already in progress: creating tablet
I20250625 07:52:33.634264 1162 raft_consensus.cc:357] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.641367 1161 tablet_replica.cc:331] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:33.641486 1162 raft_consensus.cc:383] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.641748 1161 raft_consensus.cc:2241] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.641861 1162 raft_consensus.cc:738] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.642136 1162 consensus_queue.cc:260] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.642531 1161 raft_consensus.cc:2270] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.644029 1162 ts_tablet_manager.cc:1428] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 07:52:33.644747 1161 ts_tablet_manager.cc:1905] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.645083 1162 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.649039 1163 tablet_bootstrap.cc:654] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.650550 1162 tablet_bootstrap.cc:654] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.650524 1160 ts_tablet_manager.cc:1918] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.650628 1160 log.cc:1199] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/086c9321a70e44b5bb3ea419bb466fbe
I20250625 07:52:33.650970 1160 ts_tablet_manager.cc:1939] T 086c9321a70e44b5bb3ea419bb466fbe P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.653748 2178 raft_consensus.cc:491] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.653882 2178 raft_consensus.cc:513] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.654440 2178 leader_election.cc:290] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:33.656191 1158 tablet_bootstrap.cc:492] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.656308 1158 ts_tablet_manager.cc:1397] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
W20250625 07:52:33.656865 986 leader_election.cc:343] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b1422987deca41a79f8db666d6961643
I20250625 07:52:33.657038 1158 raft_consensus.cc:357] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.657181 1158 raft_consensus.cc:383] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.657227 1158 raft_consensus.cc:738] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.657526 1158 consensus_queue.cc:260] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.657676 1161 ts_tablet_manager.cc:1918] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:33.657930 985 leader_election.cc:343] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b1422987deca41a79f8db666d6961643
I20250625 07:52:33.658020 985 leader_election.cc:304] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.658437 2178 raft_consensus.cc:2747] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.658670 1161 log.cc:1199] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3bed3f04c60249aaa389c26788974f58
I20250625 07:52:33.659003 1161 ts_tablet_manager.cc:1939] T 3bed3f04c60249aaa389c26788974f58 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:33.660288 1158 ts_tablet_manager.cc:1428] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:33.660568 1158 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.662225 1158 tablet_bootstrap.cc:654] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.663460 1162 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.663654 1162 ts_tablet_manager.cc:1397] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250625 07:52:33.664494 1162 raft_consensus.cc:357] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.665941 1162 raft_consensus.cc:383] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.666067 1162 raft_consensus.cc:738] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.666332 1162 consensus_queue.cc:260] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.666970 1162 ts_tablet_manager.cc:1428] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:33.666949 2162 consensus_queue.cc:1035] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:33.668032 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3bed3f04c60249aaa389c26788974f58 (table default.Table_a6db9072062d4e5895af0868d99ac31b [id=8e22a511cbc44f8996d35e514dfda631]) successfully deleted
I20250625 07:52:33.668540 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1123) took 1708 ms. Trace:
I20250625 07:52:33.668633 1160 rpcz_store.cc:276] 0625 07:52:31.960314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.971970 (+ 11656us) service_pool.cc:225] Handling call
0625 07:52:33.559388 (+1587418us) ts_tablet_manager.cc:1145] Deleting tablet 086c9321a70e44b5bb3ea419bb466fbe
0625 07:52:33.559396 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.587321 (+ 27925us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.650400 (+ 63079us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.668518 (+ 18118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587202}
I20250625 07:52:33.669143 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 086c9321a70e44b5bb3ea419bb466fbe (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]) successfully deleted
I20250625 07:52:33.669634 1162 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:33.673223 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 74fa6e5ac74444729645a27fc2945bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fa6e5ac74444729645a27fc2945bc6 already in progress: creating tablet
W20250625 07:52:33.674037 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a2481ca7e8db42a3a51270158bb2e2fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2481ca7e8db42a3a51270158bb2e2fa already in progress: creating tablet
I20250625 07:52:33.674409 1162 tablet_bootstrap.cc:654] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:33.675009 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f35ac535578d4681ac998a530b1f1d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f35ac535578d4681ac998a530b1f1d4f already in progress: creating tablet
W20250625 07:52:33.675765 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 68f903943e9a4151b6fa1e71d1c05473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f903943e9a4151b6fa1e71d1c05473 already in progress: creating tablet
I20250625 07:52:33.678597 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_de8edff1052842d9bd91dc30913c757b" } new_table_name: "default.Table_f94d0ba3fb39477f8178d82e056f9309" modify_external_catalogs: true
I20250625 07:52:33.678813 1160 tablet_replica.cc:331] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.679013 1160 raft_consensus.cc:2241] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.679157 1160 raft_consensus.cc:2270] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.679648 1160 ts_tablet_manager.cc:1905] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.691124 1160 ts_tablet_manager.cc:1918] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.691251 1160 log.cc:1199] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/34a338005a7b44b9b0fba7a7096cfcb8
I20250625 07:52:33.691527 1160 ts_tablet_manager.cc:1939] T 34a338005a7b44b9b0fba7a7096cfcb8 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.691893 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1126) took 1719 ms. Trace:
I20250625 07:52:33.691978 1160 rpcz_store.cc:276] 0625 07:52:31.972831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.973102 (+ 271us) service_pool.cc:225] Handling call
0625 07:52:33.678762 (+1705660us) ts_tablet_manager.cc:1145] Deleting tablet 34a338005a7b44b9b0fba7a7096cfcb8
0625 07:52:33.678773 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.682786 (+ 4013us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.689595 (+ 6809us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.691878 (+ 2283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1705487}
W20250625 07:52:33.692284 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57f763f08289455995c930d9ee03c95c consensus state has no leader
I20250625 07:52:33.692405 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 34a338005a7b44b9b0fba7a7096cfcb8 (table default.Table_3aeb5c70494240aaaad67f311ec1723b [id=2cef554f33094c77b8684986968dadef]) successfully deleted
I20250625 07:52:33.693730 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e89d1c8b956c4d12a0b715252d470852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.693931 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1122) took 1804 ms. Trace:
I20250625 07:52:33.694018 1160 rpcz_store.cc:276] 0625 07:52:31.889790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:31.988491 (+ 98701us) service_pool.cc:225] Handling call
0625 07:52:33.693790 (+1705299us) ts_tablet_manager.cc:1145] Deleting tablet 64ca83369ffe46ea9bd756f20da936cf
0625 07:52:33.693801 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.693911 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704798}
W20250625 07:52:33.694563 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 64ca83369ffe46ea9bd756f20da936cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64ca83369ffe46ea9bd756f20da936cf already in progress: creating tablet
I20250625 07:52:33.695281 1160 tablet_replica.cc:331] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.695478 1160 raft_consensus.cc:2241] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.695653 1160 raft_consensus.cc:2270] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:33.696113 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e89d1c8b956c4d12a0b715252d470852 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.696163 1160 ts_tablet_manager.cc:1905] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.699782 2131 tablet.cc:1722] T 89c703dc3249408faeb522cef0248111 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.701190 2218 tablet.cc:1722] T 89c703dc3249408faeb522cef0248111 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.718783 1160 ts_tablet_manager.cc:1918] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.718909 1160 log.cc:1199] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/110fe299c7c44df69f459da72a1138a5
I20250625 07:52:33.719210 1160 ts_tablet_manager.cc:1939] T 110fe299c7c44df69f459da72a1138a5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.719511 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1127) took 1684 ms. Trace:
I20250625 07:52:33.719522 2171 tablet.cc:1722] T 89c703dc3249408faeb522cef0248111 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:33.720119 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5b13c1f3f0e04ea3a1b5077d303e7409 (DEFAULT_TABLE table=default.Table_346480a0ddc54315b4df65b09d71294f [id=42110d4ca5d2457dbe28f9c5fbc68326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.720578 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b13c1f3f0e04ea3a1b5077d303e7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.722131 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5b13c1f3f0e04ea3a1b5077d303e7409 (DEFAULT_TABLE table=default.Table_346480a0ddc54315b4df65b09d71294f [id=42110d4ca5d2457dbe28f9c5fbc68326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.723372 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:33.723474 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f326a912662f42e2abb74225853c112d (DEFAULT_TABLE table=default.Table_5d5d75af69a841d5b3961973eb62dbab [id=323ccf97ef774c7590f972629bd5b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.723711 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:33.723948 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:33.724108 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:33.725250 894 tablet_service.cc:1468] Processing CreateTablet for tablet 5b13c1f3f0e04ea3a1b5077d303e7409 (DEFAULT_TABLE table=default.Table_346480a0ddc54315b4df65b09d71294f [id=42110d4ca5d2457dbe28f9c5fbc68326]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.725409 752 tablet_service.cc:1468] Processing CreateTablet for tablet f326a912662f42e2abb74225853c112d (DEFAULT_TABLE table=default.Table_5d5d75af69a841d5b3961973eb62dbab [id=323ccf97ef774c7590f972629bd5b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.725622 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b13c1f3f0e04ea3a1b5077d303e7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.725754 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326a912662f42e2abb74225853c112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.729839 893 tablet_service.cc:1468] Processing CreateTablet for tablet f326a912662f42e2abb74225853c112d (DEFAULT_TABLE table=default.Table_5d5d75af69a841d5b3961973eb62dbab [id=323ccf97ef774c7590f972629bd5b8e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.730193 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326a912662f42e2abb74225853c112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.730250 746 tablet_service.cc:1468] Processing CreateTablet for tablet 561910c7675b49af93bd30b93cd9a1b5 (DEFAULT_TABLE table=default.Table_59367713e2b8459791d1db51abc86d41 [id=4731c5a0f6e44d57b2d930b63d70ef9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.730566 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561910c7675b49af93bd30b93cd9a1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.730733 590 catalog_manager.cc:5582] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.731240 590 catalog_manager.cc:5582] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.731585 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]). Expected version 1 got 0
I20250625 07:52:33.731863 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:33.732085 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:33.732494 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:33.732822 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_4e9b498709a94e19a3f242fad2d3abb2 [id=29e4ffaed4f148b6b10e38231435c071] alter complete (version 1)
I20250625 07:52:33.733227 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1d2d236daca14dee8585a320d9045d88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.733775 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d2d236daca14dee8585a320d9045d88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.735563 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 561910c7675b49af93bd30b93cd9a1b5 (DEFAULT_TABLE table=default.Table_59367713e2b8459791d1db51abc86d41 [id=4731c5a0f6e44d57b2d930b63d70ef9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.745121 894 tablet_service.cc:1468] Processing CreateTablet for tablet 561910c7675b49af93bd30b93cd9a1b5 (DEFAULT_TABLE table=default.Table_59367713e2b8459791d1db51abc86d41 [id=4731c5a0f6e44d57b2d930b63d70ef9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.745551 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561910c7675b49af93bd30b93cd9a1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.746747 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d33d8ad6931344eeb198421b3645c267 (DEFAULT_TABLE table=default.Table_c261d05cf03b422fa40dc4ffdd2fb691 [id=6ce659ec2bd24b3286b794f31a75d95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.750919 756 tablet_service.cc:1468] Processing CreateTablet for tablet d33d8ad6931344eeb198421b3645c267 (DEFAULT_TABLE table=default.Table_c261d05cf03b422fa40dc4ffdd2fb691 [id=6ce659ec2bd24b3286b794f31a75d95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.751582 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d8ad6931344eeb198421b3645c267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.751607 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 034903d699584feb91f35cbb2f62701a
I20250625 07:52:33.752070 591 catalog_manager.cc:6652] ReplaceTablet: tablet 034903d699584feb91f35cbb2f62701a deleted and replaced by tablet 37d228244cca4052889acdc8f1ddd440
I20250625 07:52:33.752398 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_97b7cf4f260c44ac87961d4a83a7d50a" } new_table_name: "default.Table_77a9000fc7304886beb0d06b143debdc" modify_external_catalogs: true
I20250625 07:52:33.754542 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 034903d699584feb91f35cbb2f62701a with delete_type TABLET_DATA_DELETED (replaced by tablet 37d228244cca4052889acdc8f1ddd440) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.755252 880 tablet_service.cc:1468] Processing CreateTablet for tablet d33d8ad6931344eeb198421b3645c267 (DEFAULT_TABLE table=default.Table_c261d05cf03b422fa40dc4ffdd2fb691 [id=6ce659ec2bd24b3286b794f31a75d95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.755618 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d8ad6931344eeb198421b3645c267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.755769 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 034903d699584feb91f35cbb2f62701a with delete_type TABLET_DATA_DELETED (replaced by tablet 37d228244cca4052889acdc8f1ddd440) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.758590 1163 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.758998 1163 ts_tablet_manager.cc:1397] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.001s
I20250625 07:52:33.760794 1163 raft_consensus.cc:357] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.761076 1163 raft_consensus.cc:383] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.761197 1163 raft_consensus.cc:738] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.761360 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 0f09f9041321456e99639a27fd7d8c4c (DEFAULT_TABLE table=default.Table_0ead4f99381840579eabcd8334f44489 [id=efc5c5dc636a41fdbefb7526a0674bf6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:33.761869 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 034903d699584feb91f35cbb2f62701a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 034903d699584feb91f35cbb2f62701a already in progress: creating tablet
I20250625 07:52:33.761961 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0f09f9041321456e99639a27fd7d8c4c (DEFAULT_TABLE table=default.Table_0ead4f99381840579eabcd8334f44489 [id=efc5c5dc636a41fdbefb7526a0674bf6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:33.762173 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 034903d699584feb91f35cbb2f62701a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 034903d699584feb91f35cbb2f62701a already in progress: creating tablet
I20250625 07:52:33.762619 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f09f9041321456e99639a27fd7d8c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.765012 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 034903d699584feb91f35cbb2f62701a with delete_type TABLET_DATA_DELETED (replaced by tablet 37d228244cca4052889acdc8f1ddd440) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.776491 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 24c767b82ed24df1b145ff2df33ba155
I20250625 07:52:33.719601 1160 rpcz_store.cc:276] 0625 07:52:32.034682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.034737 (+ 55us) service_pool.cc:225] Handling call
0625 07:52:33.695237 (+1660500us) ts_tablet_manager.cc:1145] Deleting tablet 110fe299c7c44df69f459da72a1138a5
0625 07:52:33.695247 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.704014 (+ 8767us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.718668 (+ 14654us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.719494 (+ 826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660315}
I20250625 07:52:33.778338 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1128) took 1696 ms. Trace:
I20250625 07:52:33.778410 1160 rpcz_store.cc:276] 0625 07:52:32.082285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.092915 (+ 10630us) service_pool.cc:225] Handling call
0625 07:52:33.778207 (+1685292us) ts_tablet_manager.cc:1145] Deleting tablet 51c584e6bffb4f0a8020dadf78d718a7
0625 07:52:33.778217 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.778322 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1685117}
I20250625 07:52:33.778571 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c767b82ed24df1b145ff2df33ba155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.779032 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 110fe299c7c44df69f459da72a1138a5 (table default.Table_55794188bd9d47d29bc77e1c1498512f [id=d281de0430994f13a49c3d53c062dec4]) successfully deleted
I20250625 07:52:33.779130 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c767b82ed24df1b145ff2df33ba155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:33.779278 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 51c584e6bffb4f0a8020dadf78d718a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51c584e6bffb4f0a8020dadf78d718a7 already in progress: creating tablet
I20250625 07:52:33.779304 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1132) took 1682 ms. Trace:
I20250625 07:52:33.779377 1160 rpcz_store.cc:276] 0625 07:52:32.096843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.096903 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:33.778621 (+1681718us) ts_tablet_manager.cc:1145] Deleting tablet 5099f01a4a124d23a7c32accbd576e54
0625 07:52:33.778629 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.779289 (+ 660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681557}
W20250625 07:52:33.779511 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 24c767b82ed24df1b145ff2df33ba155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c767b82ed24df1b145ff2df33ba155 already in progress: creating tablet
W20250625 07:52:33.779733 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5099f01a4a124d23a7c32accbd576e54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5099f01a4a124d23a7c32accbd576e54 already in progress: creating tablet
I20250625 07:52:33.779793 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1133) took 1682 ms. Trace:
I20250625 07:52:33.779855 1160 rpcz_store.cc:276] 0625 07:52:32.097191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.097236 (+ 45us) service_pool.cc:225] Handling call
0625 07:52:33.779688 (+1682452us) ts_tablet_manager.cc:1145] Deleting tablet 3cdbdd1e3b734463ab452bafecb75822
0625 07:52:33.779697 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.779779 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682320}
W20250625 07:52:33.780258 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3cdbdd1e3b734463ab452bafecb75822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cdbdd1e3b734463ab452bafecb75822 already in progress: creating tablet
I20250625 07:52:33.780579 1160 tablet_replica.cc:331] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.780787 1160 raft_consensus.cc:2241] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.780933 1160 raft_consensus.cc:2270] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.781930 1160 ts_tablet_manager.cc:1905] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.782894 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c767b82ed24df1b145ff2df33ba155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:33.783610 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 24c767b82ed24df1b145ff2df33ba155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c767b82ed24df1b145ff2df33ba155 already in progress: creating tablet
I20250625 07:52:33.783677 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_afa358a8ea9a4e999a4160be5d7f51ed" } new_table_name: "default.Table_cb409c5e3a884ffaac6321d86f5b0e03" modify_external_catalogs: true
I20250625 07:52:33.792881 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7d42f5f3910742f0942742eb77ea1435
I20250625 07:52:33.793581 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4160b9c65d7a48fb912f32a20fed6818"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.793767 2178 raft_consensus.cc:491] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.793913 2178 raft_consensus.cc:513] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:33.794162 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4160b9c65d7a48fb912f32a20fed6818 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.794530 2178 leader_election.cc:290] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.794709 1162 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.794847 1162 ts_tablet_manager.cc:1397] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.006s
W20250625 07:52:33.795536 985 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:33.795642 1162 raft_consensus.cc:357] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:33.795737 986 leader_election.cc:343] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:33.795809 1162 raft_consensus.cc:383] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.795812 986 leader_election.cc:304] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.795861 1162 raft_consensus.cc:738] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.796036 1162 consensus_queue.cc:260] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.796466 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d42f5f3910742f0942742eb77ea1435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.796512 1162 ts_tablet_manager.cc:1428] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:33.796943 2178 raft_consensus.cc:2747] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.797554 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 953cfe2c224043eca4da735c2a5dbebc
I20250625 07:52:33.798216 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6b51e4df123348e6a217afc8bdbf1d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.798684 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b51e4df123348e6a217afc8bdbf1d66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.801355 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 953cfe2c224043eca4da735c2a5dbebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.801369 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0f09f9041321456e99639a27fd7d8c4c (DEFAULT_TABLE table=default.Table_0ead4f99381840579eabcd8334f44489 [id=efc5c5dc636a41fdbefb7526a0674bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.802278 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f09f9041321456e99639a27fd7d8c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.802853 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4aa9c46d0429418ab6f3300f000908c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.803965 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aa9c46d0429418ab6f3300f000908c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.805392 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 953cfe2c224043eca4da735c2a5dbebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.805655 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a84cc65d90764ffabafb80ab9fc5f0b8 (DEFAULT_TABLE table=default.Table_9f2a23f9d8f5431f9835acf040a4a5e5 [id=e261be50cbc74f09bf50f5008fdf209f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:33.806413 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7d42f5f3910742f0942742eb77ea1435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d42f5f3910742f0942742eb77ea1435 already in progress: creating tablet
I20250625 07:52:33.807554 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33d8ad6931344eeb198421b3645c267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.807817 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b13c1f3f0e04ea3a1b5077d303e7409. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.807978 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 953cfe2c224043eca4da735c2a5dbebc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:33.808669 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 953cfe2c224043eca4da735c2a5dbebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 953cfe2c224043eca4da735c2a5dbebc already in progress: creating tablet
I20250625 07:52:33.809110 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84cc65d90764ffabafb80ab9fc5f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.809266 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f326a912662f42e2abb74225853c112d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.812795 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d42f5f3910742f0942742eb77ea1435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.812937 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f09f9041321456e99639a27fd7d8c4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:33.813159 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 953cfe2c224043eca4da735c2a5dbebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 953cfe2c224043eca4da735c2a5dbebc already in progress: creating tablet
W20250625 07:52:33.815027 713 consensus_peers.cc:487] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:33.815323 1162 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.816807 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561910c7675b49af93bd30b93cd9a1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.816819 1163 consensus_queue.cc:260] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.817025 2261 raft_consensus.cc:491] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.817158 1162 tablet_bootstrap.cc:654] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.817145 2261 raft_consensus.cc:513] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.817907 2261 leader_election.cc:290] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.818403 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "716e0c4dab584788bbb7631b4ca934bf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:33.818864 851 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.819332 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "716e0c4dab584788bbb7631b4ca934bf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.819728 850 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.819828 850 leader_election.cc:304] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.820128 2112 raft_consensus.cc:2747] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.819633 1163 ts_tablet_manager.cc:1428] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20250625 07:52:33.820797 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d42f5f3910742f0942742eb77ea1435 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:33.821187 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7d42f5f3910742f0942742eb77ea1435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d42f5f3910742f0942742eb77ea1435 already in progress: creating tablet
I20250625 07:52:33.822417 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a7dd3b2533244b93a88d12660ea0a4b9
I20250625 07:52:33.823771 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.825323 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:33.825879 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a7dd3b2533244b93a88d12660ea0a4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7dd3b2533244b93a88d12660ea0a4b9 already in progress: creating tablet
I20250625 07:52:33.826903 2178 raft_consensus.cc:491] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.827015 2178 raft_consensus.cc:513] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.827575 2178 leader_election.cc:290] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:33.828027 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:33.828253 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a7dd3b2533244b93a88d12660ea0a4b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.828792 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02ec8c9e6340497abe802c7345f42921" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:33.829022 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a7dd3b2533244b93a88d12660ea0a4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7dd3b2533244b93a88d12660ea0a4b9 already in progress: creating tablet
I20250625 07:52:33.829243 985 leader_election.cc:304] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:33.829666 2178 raft_consensus.cc:2747] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.831995 1160 ts_tablet_manager.cc:1918] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:33.832096 1160 log.cc:1199] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/97b9a14151fc472e956126620633be32
I20250625 07:52:33.832386 1160 ts_tablet_manager.cc:1939] T 97b9a14151fc472e956126620633be32 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:33.838678 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c33b8be28b684f4babfa27e829bb64cf
I20250625 07:52:33.839900 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c9cb04ac9658429194e1997a689e4d2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.840440 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9cb04ac9658429194e1997a689e4d2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.842286 1163 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:33.842409 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_d5657e03390941c28681972134ee0159 [id=618d684f882f425dab0d9a7395deb004] alter complete (version 3)
I20250625 07:52:33.843910 1163 tablet_bootstrap.cc:654] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.844045 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c8b9e38da784424b8fbd8b3b8649a150"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.844544 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8b9e38da784424b8fbd8b3b8649a150 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.846899 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.847090 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.847288 756 tablet_service.cc:1515] Processing DeleteTablet for tablet c33b8be28b684f4babfa27e829bb64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:33.847857 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c33b8be28b684f4babfa27e829bb64cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33b8be28b684f4babfa27e829bb64cf already in progress: creating tablet
I20250625 07:52:33.848131 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c33b8be28b684f4babfa27e829bb64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:33.849573 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet c33b8be28b684f4babfa27e829bb64cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:33 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:33.849692 1162 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:33.849810 1162 ts_tablet_manager.cc:1397] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
W20250625 07:52:33.850924 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c33b8be28b684f4babfa27e829bb64cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33b8be28b684f4babfa27e829bb64cf already in progress: creating tablet
I20250625 07:52:33.851090 1162 raft_consensus.cc:357] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.851292 1162 raft_consensus.cc:383] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.851436 1162 raft_consensus.cc:738] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.851820 1162 consensus_queue.cc:260] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.852519 1162 ts_tablet_manager.cc:1428] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:33.853643 1162 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:33.865243 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 02ec8c9e6340497abe802c7345f42921 (table default.Table_965ace13b8ae452ab1e388fe37111c9a [id=eecbc272d0844eaaaa51584b812f2321]). Expected version 5 got 0
W20250625 07:52:33.865471 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4d367cc123e4bbda8054d1d61d29be1 consensus state has no leader
W20250625 07:52:33.866146 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60edc8fa184542a0b54b89736831730d consensus state has no leader
I20250625 07:52:33.871762 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]). Expected version 1 got 0
I20250625 07:52:33.872017 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]). Expected version 1 got 0
I20250625 07:52:33.872278 593 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_a9c6f935691440fe82ca615932a43240 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]). Expected version 1 got 0
I20250625 07:52:33.869853 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.870047 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.870150 749 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.870270 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.870409 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.872797 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.877923 756 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:33.880441 1162 tablet_bootstrap.cc:654] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.881174 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1139) took 1723 ms. Trace:
I20250625 07:52:33.881314 1160 rpcz_store.cc:276] 0625 07:52:32.157705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.157748 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:33.780527 (+1622779us) ts_tablet_manager.cc:1145] Deleting tablet 97b9a14151fc472e956126620633be32
0625 07:52:33.780539 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.786198 (+ 5659us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.831904 (+ 45706us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:33.881150 (+ 49246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1622579}
I20250625 07:52:33.881503 1158 tablet_bootstrap.cc:492] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.881628 1158 ts_tablet_manager.cc:1397] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.221s user 0.016s sys 0.004s
I20250625 07:52:33.881953 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 97b9a14151fc472e956126620633be32 (table default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]) successfully deleted
I20250625 07:52:33.882112 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1143) took 1636 ms. Trace:
I20250625 07:52:33.882195 1160 rpcz_store.cc:276] 0625 07:52:32.245460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.256035 (+ 10575us) service_pool.cc:225] Handling call
0625 07:52:33.881988 (+1625953us) ts_tablet_manager.cc:1145] Deleting tablet bd9a6cded85c4d6296f8468644de2016
0625 07:52:33.881997 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.882095 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625699}
W20250625 07:52:33.882649 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bd9a6cded85c4d6296f8468644de2016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd9a6cded85c4d6296f8468644de2016 already in progress: creating tablet
I20250625 07:52:33.882964 1158 raft_consensus.cc:357] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.883164 1158 raft_consensus.cc:383] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.883225 1158 raft_consensus.cc:738] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.883404 1160 tablet_replica.cc:331] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:33.883466 1158 consensus_queue.cc:260] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.883822 1160 raft_consensus.cc:2241] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:33.884706 1160 raft_consensus.cc:2270] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.884773 1166 tablet_replica.cc:331] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:33.884956 1166 raft_consensus.cc:2241] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:33.885156 1166 pending_rounds.cc:70] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:33.885192 1166 pending_rounds.cc:77] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432257412046848 op_type: NO_OP noop_request { }
I20250625 07:52:33.885267 1166 raft_consensus.cc:2887] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:33.885332 1166 raft_consensus.cc:2270] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.885442 1160 ts_tablet_manager.cc:1905] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.886154 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aac4a06e92064896958276d8b9438ec0" } new_table_name: "default.Table_203e763b155e44df936ef87fdc585bad" modify_external_catalogs: true
I20250625 07:52:33.887450 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ce8a2c36677445d2bab1c65eb72ffc37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:33.887887 1166 ts_tablet_manager.cc:1905] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:33.888021 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce8a2c36677445d2bab1c65eb72ffc37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.889187 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_364620262bdb4f57bb3be61e6f9aa395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.889590 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_364620262bdb4f57bb3be61e6f9aa395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.893390 1158 ts_tablet_manager.cc:1428] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:33.893534 2261 raft_consensus.cc:491] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.893668 2261 raft_consensus.cc:513] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.894286 2261 leader_election.cc:290] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.894469 1166 ts_tablet_manager.cc:1918] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:33.894555 1166 log.cc:1199] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/34a338005a7b44b9b0fba7a7096cfcb8
I20250625 07:52:33.894860 1166 ts_tablet_manager.cc:1939] T 34a338005a7b44b9b0fba7a7096cfcb8 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:33.894819 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:33.895328 851 leader_election.cc:343] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.895823 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 34a338005a7b44b9b0fba7a7096cfcb8 (table default.Table_3aeb5c70494240aaaad67f311ec1723b [id=2cef554f33094c77b8684986968dadef]) successfully deleted
I20250625 07:52:33.896270 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:33.896870 1166 tablet_replica.cc:331] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:33.897096 1166 raft_consensus.cc:2241] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:33.897341 1166 pending_rounds.cc:70] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:33.897382 1163 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:33.897502 1163 ts_tablet_manager.cc:1397] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250625 07:52:33.898339 1163 raft_consensus.cc:357] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.898515 1158 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:33.898519 1163 raft_consensus.cc:383] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.898574 1163 raft_consensus.cc:738] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.898793 1163 consensus_queue.cc:260] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:33.900202 1158 tablet_bootstrap.cc:654] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.900677 2045 raft_consensus.cc:491] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.901173 2045 raft_consensus.cc:513] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.901266 1163 ts_tablet_manager.cc:1428] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:33.901865 2045 leader_election.cc:290] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:33.902259 850 leader_election.cc:343] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.902344 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:33.902544 911 raft_consensus.cc:2466] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:33.902801 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:33.902360 850 leader_election.cc:304] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.903854 713 leader_election.cc:304] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:33.904152 2261 raft_consensus.cc:2747] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.904650 2045 raft_consensus.cc:2802] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:33.904747 2045 raft_consensus.cc:491] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:33.904800 2045 raft_consensus.cc:3058] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:33.905037 713 leader_election.cc:343] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.905563 756 tablet_service.cc:1468] Processing CreateTablet for tablet a84cc65d90764ffabafb80ab9fc5f0b8 (DEFAULT_TABLE table=default.Table_9f2a23f9d8f5431f9835acf040a4a5e5 [id=e261be50cbc74f09bf50f5008fdf209f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.905936 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84cc65d90764ffabafb80ab9fc5f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.907392 2045 raft_consensus.cc:513] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.910657 752 tablet_service.cc:1468] Processing CreateTablet for tablet 570e224ff3a94e8d84064166ef76b838 (DEFAULT_TABLE table=default.Table_6208679543004885b54f1f20d7f98f9d [id=779bdabec16a4ff8a501569104e03dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.910657 746 tablet_service.cc:1468] Processing CreateTablet for tablet b990b9f98e88478a9c79b29c6a8f07c5 (DEFAULT_TABLE table=default.Table_5df78364c7b841979d418ddeecc476b7 [id=495c051f725b4d7d93088698807a0260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.911137 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570e224ff3a94e8d84064166ef76b838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.911299 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b990b9f98e88478a9c79b29c6a8f07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.914222 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f31104720614b028e373ae1500c31cc from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:33.914352 741 tablet_service.cc:1468] Processing CreateTablet for tablet f1a22d57d846414a8fa5855dc542b594 (DEFAULT_TABLE table=default.Table_5c1c689b7e634891a7e82c704c095684 [id=1ec4fcf344f545f1af25ef8e7c5d4a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.914655 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a22d57d846414a8fa5855dc542b594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.914786 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aab39ebeac1742cda147feb57cf092d7" } new_table_name: "default.Table_2abbb2dbe82c49db85360a028645c2bd" modify_external_catalogs: true
I20250625 07:52:33.916201 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1c2cf30b339244f39bef84e1af811bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:33.916674 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c2cf30b339244f39bef84e1af811bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:33.916944 749 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7d404c595f4782ab13a3a847e0e40e (DEFAULT_TABLE table=default.Table_9c52eceef6e74723abcc7fce45c50f1e [id=30848f22ce0d4a43bb13ce853c134afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.917245 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7d404c595f4782ab13a3a847e0e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.918007 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_744b58275931414c9463ddc96dfe14aa" } new_table_name: "default.Table_88c489ee92a349e69319d410220e7110" modify_external_catalogs: true
I20250625 07:52:33.920470 1011 tablet_service.cc:1468] Processing CreateTablet for tablet b990b9f98e88478a9c79b29c6a8f07c5 (DEFAULT_TABLE table=default.Table_5df78364c7b841979d418ddeecc476b7 [id=495c051f725b4d7d93088698807a0260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.920864 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b990b9f98e88478a9c79b29c6a8f07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.920892 758 tablet_service.cc:1468] Processing CreateTablet for tablet 00694ce90a484f00b1d9eb2d6fa8ec12 (DEFAULT_TABLE table=default.Table_7d69398f95854f01909ebfe2310d23f0 [id=e3653422c5714cd59f37566313574548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.920917 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 00694ce90a484f00b1d9eb2d6fa8ec12 (DEFAULT_TABLE table=default.Table_7d69398f95854f01909ebfe2310d23f0 [id=e3653422c5714cd59f37566313574548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.921065 1027 tablet_service.cc:1468] Processing CreateTablet for tablet bf6e5b6f54714ec5b675f3a50af32417 (DEFAULT_TABLE table=default.Table_0712d103f5094acb959ac7e976036190 [id=75befa9bba0a484b95311e933670c9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.921178 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00694ce90a484f00b1d9eb2d6fa8ec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.921188 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00694ce90a484f00b1d9eb2d6fa8ec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.921396 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6e5b6f54714ec5b675f3a50af32417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.921422 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 (DEFAULT_TABLE table=default.Table_77a9000fc7304886beb0d06b143debdc [id=c55dda8057d5429f9c4327ed8b9b8e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.921818 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d367cc123e4bbda8054d1d61d29be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.924916 754 tablet_service.cc:1468] Processing CreateTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 (DEFAULT_TABLE table=default.Table_77a9000fc7304886beb0d06b143debdc [id=c55dda8057d5429f9c4327ed8b9b8e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.925035 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f1a22d57d846414a8fa5855dc542b594 (DEFAULT_TABLE table=default.Table_5c1c689b7e634891a7e82c704c095684 [id=1ec4fcf344f545f1af25ef8e7c5d4a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.925191 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d367cc123e4bbda8054d1d61d29be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.925341 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a22d57d846414a8fa5855dc542b594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.925756 876 tablet_service.cc:1468] Processing CreateTablet for tablet 00694ce90a484f00b1d9eb2d6fa8ec12 (DEFAULT_TABLE table=default.Table_7d69398f95854f01909ebfe2310d23f0 [id=e3653422c5714cd59f37566313574548]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.925798 894 tablet_service.cc:1468] Processing CreateTablet for tablet b990b9f98e88478a9c79b29c6a8f07c5 (DEFAULT_TABLE table=default.Table_5df78364c7b841979d418ddeecc476b7 [id=495c051f725b4d7d93088698807a0260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.926151 880 tablet_service.cc:1468] Processing CreateTablet for tablet 570e224ff3a94e8d84064166ef76b838 (DEFAULT_TABLE table=default.Table_6208679543004885b54f1f20d7f98f9d [id=779bdabec16a4ff8a501569104e03dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.926453 885 tablet_service.cc:1468] Processing CreateTablet for tablet f1a22d57d846414a8fa5855dc542b594 (DEFAULT_TABLE table=default.Table_5c1c689b7e634891a7e82c704c095684 [id=1ec4fcf344f545f1af25ef8e7c5d4a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.926766 890 tablet_service.cc:1468] Processing CreateTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 (DEFAULT_TABLE table=default.Table_77a9000fc7304886beb0d06b143debdc [id=c55dda8057d5429f9c4327ed8b9b8e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.927095 893 tablet_service.cc:1468] Processing CreateTablet for tablet a84cc65d90764ffabafb80ab9fc5f0b8 (DEFAULT_TABLE table=default.Table_9f2a23f9d8f5431f9835acf040a4a5e5 [id=e261be50cbc74f09bf50f5008fdf209f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.927855 752 tablet_service.cc:1468] Processing CreateTablet for tablet bf6e5b6f54714ec5b675f3a50af32417 (DEFAULT_TABLE table=default.Table_0712d103f5094acb959ac7e976036190 [id=75befa9bba0a484b95311e933670c9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.928138 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6e5b6f54714ec5b675f3a50af32417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.928931 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7d404c595f4782ab13a3a847e0e40e (DEFAULT_TABLE table=default.Table_9c52eceef6e74723abcc7fce45c50f1e [id=30848f22ce0d4a43bb13ce853c134afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.928925 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7d404c595f4782ab13a3a847e0e40e (DEFAULT_TABLE table=default.Table_9c52eceef6e74723abcc7fce45c50f1e [id=30848f22ce0d4a43bb13ce853c134afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.929294 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7d404c595f4782ab13a3a847e0e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.929970 595 catalog_manager.cc:5582] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.930369 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_a23788a3c209434db9f0ef44e0220672 [id=c80e78f1ef124af2b8f405e7d162ade8]). Expected version 2 got 0
I20250625 07:52:33.930612 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet cffde34fbba34e44b67985d3480666c3 (table default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb]). Expected version 1 got 0
I20250625 07:52:33.930914 595 catalog_manager.cc:5582] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:33.931351 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:33.931550 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]). Expected version 1 got 0
I20250625 07:52:33.931885 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]). Expected version 1 got 0
I20250625 07:52:33.933107 2045 leader_election.cc:290] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.933121 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 570e224ff3a94e8d84064166ef76b838 (DEFAULT_TABLE table=default.Table_6208679543004885b54f1f20d7f98f9d [id=779bdabec16a4ff8a501569104e03dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.933462 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570e224ff3a94e8d84064166ef76b838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.933823 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:33.933985 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:33.934135 911 raft_consensus.cc:3058] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:33.937211 713 leader_election.cc:343] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.937577 911 raft_consensus.cc:2466] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:33.937945 713 leader_election.cc:304] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.938490 2161 raft_consensus.cc:2802] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:33.938593 2161 raft_consensus.cc:695] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:33.938817 2161 consensus_queue.cc:237] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.897477 1166 pending_rounds.cc:77] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432255532294144 op_type: NO_OP noop_request { }
I20250625 07:52:33.945283 1166 raft_consensus.cc:2887] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:33.945500 1166 raft_consensus.cc:2270] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:33.954209 1166 ts_tablet_manager.cc:1905] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:33.963143 1158 tablet_bootstrap.cc:492] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:33.963419 1158 ts_tablet_manager.cc:1397] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.004s
I20250625 07:52:33.964408 1158 raft_consensus.cc:357] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.968840 2045 raft_consensus.cc:491] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:33.969038 2045 raft_consensus.cc:513] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.969743 2045 leader_election.cc:290] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:33.971685 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "110fe299c7c44df69f459da72a1138a5" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:33.973157 713 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 110fe299c7c44df69f459da72a1138a5
W20250625 07:52:33.973349 713 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:33.973445 713 leader_election.cc:304] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:33.974143 2045 raft_consensus.cc:2747] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:33.975075 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4d11294b8fdd42c5aaaf0e271863258a (DEFAULT_TABLE table=default.Table_b36ed46abbe249a897381bd2b81d9d12 [id=504a78ec9443489e9bbc407bbdd37406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.975097 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 9fbb67d7c54b4b96a420883a43bddbe4 (DEFAULT_TABLE table=default.Table_84bea214164245c8a5debd8bd9d38fb7 [id=510733b0a2d7482fa8f12d4f559de543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.975356 877 tablet_service.cc:1468] Processing CreateTablet for tablet bf6e5b6f54714ec5b675f3a50af32417 (DEFAULT_TABLE table=default.Table_0712d103f5094acb959ac7e976036190 [id=75befa9bba0a484b95311e933670c9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.975584 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbb67d7c54b4b96a420883a43bddbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.975605 756 tablet_service.cc:1468] Processing CreateTablet for tablet 9fbb67d7c54b4b96a420883a43bddbe4 (DEFAULT_TABLE table=default.Table_84bea214164245c8a5debd8bd9d38fb7 [id=510733b0a2d7482fa8f12d4f559de543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.975585 879 tablet_service.cc:1468] Processing CreateTablet for tablet 9fbb67d7c54b4b96a420883a43bddbe4 (DEFAULT_TABLE table=default.Table_84bea214164245c8a5debd8bd9d38fb7 [id=510733b0a2d7482fa8f12d4f559de543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.975941 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4d11294b8fdd42c5aaaf0e271863258a (DEFAULT_TABLE table=default.Table_b36ed46abbe249a897381bd2b81d9d12 [id=504a78ec9443489e9bbc407bbdd37406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.976660 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4d11294b8fdd42c5aaaf0e271863258a (DEFAULT_TABLE table=default.Table_b36ed46abbe249a897381bd2b81d9d12 [id=504a78ec9443489e9bbc407bbdd37406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.977063 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d11294b8fdd42c5aaaf0e271863258a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.980703 1166 ts_tablet_manager.cc:1918] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:33.980827 1166 log.cc:1199] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/086c9321a70e44b5bb3ea419bb466fbe
I20250625 07:52:33.980968 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 (DEFAULT_TABLE table=default.Table_1e4d42ab40de4520ade73a572a5334dd [id=50bb612a94a741c486f931bfa70f9686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:33.981177 1166 ts_tablet_manager.cc:1939] T 086c9321a70e44b5bb3ea419bb466fbe P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:33.981338 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0009ea8aaf6e4fdd8ff602bd59968de3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:33.982187 713 consensus_peers.cc:487] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:33.982973 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00694ce90a484f00b1d9eb2d6fa8ec12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.982987 1163 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:33.983516 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a22d57d846414a8fa5855dc542b594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.984812 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4d367cc123e4bbda8054d1d61d29be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.985204 1163 tablet_bootstrap.cc:654] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:33.988742 1158 raft_consensus.cc:383] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:33.988818 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570e224ff3a94e8d84064166ef76b838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.988842 1158 raft_consensus.cc:738] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:33.989071 1158 consensus_queue.cc:260] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:33.992830 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b990b9f98e88478a9c79b29c6a8f07c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:33.995088 1158 ts_tablet_manager.cc:1428] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250625 07:52:34.005642 1158 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.006033 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d11294b8fdd42c5aaaf0e271863258a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.007222 1158 tablet_bootstrap.cc:654] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.008919 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbb67d7c54b4b96a420883a43bddbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.009054 1166 tablet_replica.cc:331] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.009225 1166 raft_consensus.cc:2241] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.009370 1166 raft_consensus.cc:2270] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.009840 1166 ts_tablet_manager.cc:1905] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.012943 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 086c9321a70e44b5bb3ea419bb466fbe (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]) successfully deleted
I20250625 07:52:34.013875 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 (DEFAULT_TABLE table=default.Table_1e4d42ab40de4520ade73a572a5334dd [id=50bb612a94a741c486f931bfa70f9686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.013993 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 (DEFAULT_TABLE table=default.Table_1e4d42ab40de4520ade73a572a5334dd [id=50bb612a94a741c486f931bfa70f9686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.014324 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 8a1b018bdb29404e8de21b07e9fe5817 (DEFAULT_TABLE table=default.Table_a5179ed8694740ada7407f92a210c71a [id=493e7887da654979ae38cec754e32e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.014465 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0009ea8aaf6e4fdd8ff602bd59968de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.016368 2261 raft_consensus.cc:491] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.016404 1166 ts_tablet_manager.cc:1918] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.016492 1166 log.cc:1199] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:34.016508 2261 raft_consensus.cc:513] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.016804 1166 ts_tablet_manager.cc:1939] T faf049714a7e4d7eb947c29f272bbb7c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.017167 2261 leader_election.cc:290] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.017297 1166 tablet_replica.cc:331] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.017465 1166 raft_consensus.cc:2241] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.017557 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a84cc65d90764ffabafb80ab9fc5f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.017598 1166 raft_consensus.cc:2270] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.017680 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aeb7077591f466eb77e3096f6b1d74b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.018077 1166 ts_tablet_manager.cc:1905] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.018111 851 leader_election.cc:343] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.018805 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0aeb7077591f466eb77e3096f6b1d74b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.019263 850 leader_election.cc:343] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.019353 850 leader_election.cc:304] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.019727 2261 raft_consensus.cc:2747] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.020912 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7d404c595f4782ab13a3a847e0e40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.023273 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet faf049714a7e4d7eb947c29f272bbb7c (table default.Table_4c6dba196ca04a6ba70a9180b32eeec3 [id=a1e6b3ecd3bb49ef94da294bb866f824]) successfully deleted
I20250625 07:52:34.023928 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8a1b018bdb29404e8de21b07e9fe5817 (DEFAULT_TABLE table=default.Table_a5179ed8694740ada7407f92a210c71a [id=493e7887da654979ae38cec754e32e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.024226 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1b018bdb29404e8de21b07e9fe5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.024928 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6e5b6f54714ec5b675f3a50af32417. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.027202 754 tablet_service.cc:1468] Processing CreateTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 (DEFAULT_TABLE table=default.Table_303504fb62ac468c8985d6d1b3f25851 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.027545 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387934e5204a3e9ce6f13d1dc3f777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.028930 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbb67d7c54b4b96a420883a43bddbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.032230 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1b018bdb29404e8de21b07e9fe5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.032902 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d11294b8fdd42c5aaaf0e271863258a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.038040 876 tablet_service.cc:1468] Processing CreateTablet for tablet 8a1b018bdb29404e8de21b07e9fe5817 (DEFAULT_TABLE table=default.Table_a5179ed8694740ada7407f92a210c71a [id=493e7887da654979ae38cec754e32e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.038118 888 tablet_service.cc:1468] Processing CreateTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 (DEFAULT_TABLE table=default.Table_303504fb62ac468c8985d6d1b3f25851 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.038388 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a1b018bdb29404e8de21b07e9fe5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.041170 2261 raft_consensus.cc:491] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.041291 2261 raft_consensus.cc:513] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.041365 1160 ts_tablet_manager.cc:1918] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.041465 1160 log.cc:1199] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:34.041718 1160 ts_tablet_manager.cc:1939] T af8c114c1c464bcc95dbd287c88976d6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.041847 2261 leader_election.cc:290] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.042352 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.042145 1011 tablet_service.cc:1468] Processing CreateTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 (DEFAULT_TABLE table=default.Table_303504fb62ac468c8985d6d1b3f25851 [id=9511cff29b2441689b47351734929304]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:34.042889 851 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.043084 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387934e5204a3e9ce6f13d1dc3f777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.047236 1163 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.047820 1163 ts_tablet_manager.cc:1397] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.007s
I20250625 07:52:34.048928 893 tablet_service.cc:1468] Processing CreateTablet for tablet 667e71a1cfbf4b9cad51382d7e12f633 (DEFAULT_TABLE table=default.Table_4e553aa8d6d4472ba0f5f9124ff6f580 [id=98ce65ed1da945809207e1ffcea9a046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.050514 1162 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.050781 1850 raft_consensus.cc:491] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.050844 1162 ts_tablet_manager.cc:1397] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.197s user 0.001s sys 0.007s
I20250625 07:52:34.050920 1850 raft_consensus.cc:513] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.051338 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667e71a1cfbf4b9cad51382d7e12f633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.051484 1850 leader_election.cc:290] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.051635 1162 raft_consensus.cc:357] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.051946 1162 raft_consensus.cc:383] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.052000 1162 raft_consensus.cc:738] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.052175 1162 consensus_queue.cc:260] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.043664 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.053261 1162 ts_tablet_manager.cc:1428] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:34.053530 1162 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.048554 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 (DEFAULT_TABLE table=default.Table_3f799c3e479a406690670e68e70295cf [id=38d760fd45b040d49b45af3bcc535cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.052174 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.053844 850 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.053946 850 leader_election.cc:304] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.054266 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8893ee33bd4f88ac5c913e99fb4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.054669 2261 raft_consensus.cc:2747] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.055089 713 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.055202 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:34.055338 911 raft_consensus.cc:2466] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:34.055480 1162 tablet_bootstrap.cc:654] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.056819 713 leader_election.cc:304] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.057008 1166 ts_tablet_manager.cc:1918] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.057044 1850 raft_consensus.cc:2802] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:34.057164 1850 raft_consensus.cc:491] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:34.057248 1850 raft_consensus.cc:3058] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.057497 1163 raft_consensus.cc:357] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.057646 1163 raft_consensus.cc:383] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.057685 1163 raft_consensus.cc:738] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.057171 1166 log.cc:1199] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/547b758c2e6c4558bb232f14b37aa81a
I20250625 07:52:34.058063 1163 consensus_queue.cc:260] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.058182 1166 ts_tablet_manager.cc:1939] T 547b758c2e6c4558bb232f14b37aa81a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.058601 1166 tablet_replica.cc:331] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.058885 1166 raft_consensus.cc:2241] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.058934 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 547b758c2e6c4558bb232f14b37aa81a (table default.Table_0c0f428f2b54479a8e0bd1f36ef2dede [id=83522331e5df4bafaa65c81be08c7525]) successfully deleted
I20250625 07:52:34.059037 1166 raft_consensus.cc:2270] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.059377 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 (DEFAULT_TABLE table=default.Table_3f799c3e479a406690670e68e70295cf [id=38d760fd45b040d49b45af3bcc535cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.059512 1166 ts_tablet_manager.cc:1905] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.059741 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8893ee33bd4f88ac5c913e99fb4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.060015 1850 raft_consensus.cc:513] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.052968 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 667e71a1cfbf4b9cad51382d7e12f633 (DEFAULT_TABLE table=default.Table_4e553aa8d6d4472ba0f5f9124ff6f580 [id=98ce65ed1da945809207e1ffcea9a046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.060627 1850 leader_election.cc:290] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.060732 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667e71a1cfbf4b9cad51382d7e12f633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.061573 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:34.062526 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf175e17865e40ceb26aa955b91c2c13" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:34.062670 911 raft_consensus.cc:3058] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.062852 1163 ts_tablet_manager.cc:1428] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250625 07:52:34.063122 1163 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.065562 1163 tablet_bootstrap.cc:654] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.065798 911 raft_consensus.cc:2466] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:34.062260 713 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.069479 713 leader_election.cc:304] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.069797 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0009ea8aaf6e4fdd8ff602bd59968de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.069797 2045 raft_consensus.cc:2802] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:34.070274 2045 raft_consensus.cc:695] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:34.070572 2045 consensus_queue.cc:237] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.071262 1166 ts_tablet_manager.cc:1918] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.071359 1166 log.cc:1199] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:34.071651 1166 ts_tablet_manager.cc:1939] T d393a85f5fde4b3fb57e5b3ec372791c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.072067 1166 tablet_replica.cc:331] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.072237 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet d393a85f5fde4b3fb57e5b3ec372791c (table default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]) successfully deleted
I20250625 07:52:34.072875 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc387934e5204a3e9ce6f13d1dc3f777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.073573 879 tablet_service.cc:1468] Processing CreateTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 (DEFAULT_TABLE table=default.Table_3f799c3e479a406690670e68e70295cf [id=38d760fd45b040d49b45af3bcc535cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.073899 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8893ee33bd4f88ac5c913e99fb4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.077086 1046 raft_consensus.cc:3058] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.089311 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:34.092669 2261 raft_consensus.cc:491] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.092828 2261 raft_consensus.cc:513] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.093372 2261 leader_election.cc:290] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.094363 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db12504a1a54add920561dae27d2d2b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.095016 850 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.096927 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1144) took 1752 ms. Trace:
I20250625 07:52:34.101599 1160 rpcz_store.cc:276] 0625 07:52:32.344494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.345560 (+ 1066us) service_pool.cc:225] Handling call
0625 07:52:33.883360 (+1537800us) ts_tablet_manager.cc:1145] Deleting tablet af8c114c1c464bcc95dbd287c88976d6
0625 07:52:33.883371 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:33.890250 (+ 6879us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.041290 (+151040us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.096908 (+ 55618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536753}
I20250625 07:52:34.102958 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet af8c114c1c464bcc95dbd287c88976d6 (table default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]) successfully deleted
I20250625 07:52:34.103125 1160 tablet_replica.cc:331] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.103313 1160 raft_consensus.cc:2241] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.103488 1160 raft_consensus.cc:2270] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.104032 1160 ts_tablet_manager.cc:1905] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.107846 2218 tablet.cc:1722] T 02ec8c9e6340497abe802c7345f42921 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:34.108582 1163 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.108847 1163 ts_tablet_manager.cc:1397] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20250625 07:52:34.110400 1163 raft_consensus.cc:357] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.110651 1163 raft_consensus.cc:383] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.110752 1163 raft_consensus.cc:738] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.111022 1163 consensus_queue.cc:260] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.113642 1163 ts_tablet_manager.cc:1428] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:34.113940 1163 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.117094 1163 tablet_bootstrap.cc:654] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.073158 2162 consensus_queue.cc:1035] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:34.073163 756 tablet_service.cc:1468] Processing CreateTablet for tablet 667e71a1cfbf4b9cad51382d7e12f633 (DEFAULT_TABLE table=default.Table_4e553aa8d6d4472ba0f5f9124ff6f580 [id=98ce65ed1da945809207e1ffcea9a046]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:34.125190 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667e71a1cfbf4b9cad51382d7e12f633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:34.094802 1850 raft_consensus.cc:491] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.124634 1166 raft_consensus.cc:2241] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.129277 597 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_55f91bee59d14d0a9744c1eccf96fc29 [id=abe0656f5205429c8f5dbb11d14f0353] alter complete (version 1)
I20250625 07:52:34.129349 1166 raft_consensus.cc:2270] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.129858 1166 ts_tablet_manager.cc:1905] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.131174 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58] alter complete (version 3)
I20250625 07:52:34.132741 1850 raft_consensus.cc:513] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.133311 1850 leader_election.cc:290] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.136051 1166 ts_tablet_manager.cc:1918] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.136147 1166 log.cc:1199] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:34.136397 1166 ts_tablet_manager.cc:1939] T 4167282133f94bfb9870bd08fc6594b4 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.141561 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6239713a586d43c28c101ba2447d579d with delete_type TABLET_DATA_DELETED (replaced by tablet 0657163b9a0d4f7f99f57f7fd1f32462) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.141788 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f566cc9cdd447798a5cb408ef9cd35a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.141928 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet b1422987deca41a79f8db666d6961643 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a5ff005b424ae5ae9831263abf58ef) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.142084 1161 tablet_replica.cc:331] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:34.142246 1161 raft_consensus.cc:2241] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.142383 1161 raft_consensus.cc:2270] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.142891 1161 ts_tablet_manager.cc:1905] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.144538 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c8c015c202544914b74f1e245b496f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.145565 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet fcc3afc210a540058294121d2b83c66d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.146307 1166 tablet_replica.cc:331] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.147136 1166 raft_consensus.cc:2241] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.147555 1166 raft_consensus.cc:2270] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.148248 1166 ts_tablet_manager.cc:1905] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.151594 1161 ts_tablet_manager.cc:1918] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.151758 1161 log.cc:1199] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6239713a586d43c28c101ba2447d579d
I20250625 07:52:34.152182 1161 ts_tablet_manager.cc:1939] T 6239713a586d43c28c101ba2447d579d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:34.152850 1161 tablet_replica.cc:331] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:34.152889 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6239713a586d43c28c101ba2447d579d (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
I20250625 07:52:34.153182 1161 raft_consensus.cc:2241] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.153359 1161 raft_consensus.cc:2270] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.153594 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d76681f162074b67a6b886887b5a3d61 with delete_type TABLET_DATA_DELETED (replaced by tablet f1983df4902f48d2b408a99b440368e7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.155608 1166 ts_tablet_manager.cc:1918] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.155759 1166 log.cc:1199] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/bbe5c37fc9bd49c685de8f469a7af0c6
I20250625 07:52:34.156044 1166 ts_tablet_manager.cc:1939] T bbe5c37fc9bd49c685de8f469a7af0c6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.161567 1161 ts_tablet_manager.cc:1905] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.161475 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 51172.9us
I20250625 07:52:34.162492 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db12504a1a54add920561dae27d2d2b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:34.163126 851 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.163257 851 leader_election.cc:304] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.163673 2112 raft_consensus.cc:2747] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.167330 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e5a207d515449e298936d8bb532437a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.176419 2045 raft_consensus.cc:491] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.168918 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e4bf78218f4b82b4f16f59b1704256 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.176829 2045 raft_consensus.cc:513] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.178020 2045 leader_election.cc:290] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.186295 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4bb74391a1c4e2792226d259a07f977" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.186988 713 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6e5a207d515449e298936d8bb532437a
W20250625 07:52:34.187976 713 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f4bb74391a1c4e2792226d259a07f977
W20250625 07:52:34.188751 713 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.190109 1163 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.190296 1162 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.190295 1163 ts_tablet_manager.cc:1397] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.001s
I20250625 07:52:34.190409 1162 ts_tablet_manager.cc:1397] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.137s user 0.010s sys 0.019s
I20250625 07:52:34.191061 1162 raft_consensus.cc:357] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.191064 1163 raft_consensus.cc:357] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.191211 1162 raft_consensus.cc:383] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.191231 1163 raft_consensus.cc:383] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.191265 1162 raft_consensus.cc:738] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.191280 1163 raft_consensus.cc:738] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.191452 1162 consensus_queue.cc:260] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.189105 713 leader_election.cc:304] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.191987 1162 ts_tablet_manager.cc:1428] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:34.191610 1163 consensus_queue.cc:260] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.192771 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 4167282133f94bfb9870bd08fc6594b4 (table default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]) successfully deleted
I20250625 07:52:34.192881 1163 ts_tablet_manager.cc:1428] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:34.193137 1163 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.199034 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4160b9c65d7a48fb912f32a20fed6818" } modify_external_catalogs: true
I20250625 07:52:34.201967 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet bbe5c37fc9bd49c685de8f469a7af0c6 (table default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]) successfully deleted
I20250625 07:52:34.202279 1163 tablet_bootstrap.cc:654] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.205734 1158 tablet_bootstrap.cc:492] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.206301 1158 ts_tablet_manager.cc:1397] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.201s user 0.004s sys 0.003s
I20250625 07:52:34.208068 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069] alter complete (version 1)
I20250625 07:52:34.211280 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5428a0666d7a4a21be162dd87b4348d8" } modify_external_catalogs: true
I20250625 07:52:34.211674 879 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.212069 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.212469 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.213989 1158 raft_consensus.cc:357] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.221398 2112 raft_consensus.cc:491] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.221555 2112 raft_consensus.cc:513] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.222206 2112 leader_election.cc:290] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.222929 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.224018 1163 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:34.224210 850 leader_election.cc:343] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.225903 1163 ts_tablet_manager.cc:1397] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250625 07:52:34.221161 1158 raft_consensus.cc:383] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.227725 1158 raft_consensus.cc:738] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.230252 1163 raft_consensus.cc:357] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.230780 1163 raft_consensus.cc:383] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.235496 1163 raft_consensus.cc:738] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.229887 1158 consensus_queue.cc:260] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.236207 1163 consensus_queue.cc:260] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.237051 1163 ts_tablet_manager.cc:1428] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 07:52:34.239521 1158 ts_tablet_manager.cc:1428] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.033s user 0.001s sys 0.002s
I20250625 07:52:34.241418 1158 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.242355 1163 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.244117 1163 tablet_bootstrap.cc:654] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.255383 1158 tablet_bootstrap.cc:654] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.256520 1163 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.256744 1163 ts_tablet_manager.cc:1397] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20250625 07:52:34.257614 1163 raft_consensus.cc:357] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.258392 1163 raft_consensus.cc:383] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.258554 1163 raft_consensus.cc:738] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.258877 1163 consensus_queue.cc:260] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.259452 1163 ts_tablet_manager.cc:1428] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:34.259788 1163 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.261493 1163 tablet_bootstrap.cc:654] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.263984 1158 tablet_bootstrap.cc:492] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.264079 961 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:34.264235 1158 ts_tablet_manager.cc:1397] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20250625 07:52:34.265027 1158 raft_consensus.cc:357] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.265218 1158 raft_consensus.cc:383] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.265292 1158 raft_consensus.cc:738] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.265537 1158 consensus_queue.cc:260] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.266027 1158 ts_tablet_manager.cc:1428] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.266301 1158 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.267720 1158 tablet_bootstrap.cc:654] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.274358 2112 raft_consensus.cc:491] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.274598 2112 raft_consensus.cc:513] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.275292 2112 leader_election.cc:290] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.275835 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.276298 850 leader_election.cc:343] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.277160 1163 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.277310 1163 ts_tablet_manager.cc:1397] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.003s
I20250625 07:52:34.278090 1163 raft_consensus.cc:357] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.278272 1163 raft_consensus.cc:383] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.278352 1163 raft_consensus.cc:738] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.278613 1163 consensus_queue.cc:260] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.279132 1163 ts_tablet_manager.cc:1428] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:34.279960 1163 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.281682 1163 tablet_bootstrap.cc:654] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.302945 1158 tablet_bootstrap.cc:492] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.303205 1158 ts_tablet_manager.cc:1397] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.004s
I20250625 07:52:34.303992 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4160b9c65d7a48fb912f32a20fed6818" } modify_external_catalogs: true
I20250625 07:52:34.304039 1158 raft_consensus.cc:357] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.304195 1158 raft_consensus.cc:383] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.304214 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5428a0666d7a4a21be162dd87b4348d8" } modify_external_catalogs: true
I20250625 07:52:34.304368 1158 raft_consensus.cc:738] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:34.304109 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eba09cd15d9f4611ab14d4a6dc372719 consensus state has no leader
I20250625 07:52:34.304672 1158 consensus_queue.cc:260] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.305066 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_77d400c9d724449c8fb62b8761fb2784" } modify_external_catalogs: true
I20250625 07:52:34.305148 1158 ts_tablet_manager.cc:1428] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:34.305258 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_77d400c9d724449c8fb62b8761fb2784" } modify_external_catalogs: true
I20250625 07:52:34.307490 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_213cb55216834eb4a87e67aff98467f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.307579 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:34.307792 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_4d062bccb2764b73a210a359c367f0cb [id=45c3c59676b0417fb034a9ffd06fee91]). Expected version 1 got 0
W20250625 07:52:34.308104 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213cb55216834eb4a87e67aff98467f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.308238 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:34.308676 1158 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:34.309437 591 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 2db37ece7a964f47911fe4fc7434cd07 consensus state has no leader
W20250625 07:52:34.309828 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 387a69222770408598184411225340e7 consensus state has no leader
I20250625 07:52:34.310232 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_303504fb62ac468c8985d6d1b3f25851" } modify_external_catalogs: true
I20250625 07:52:34.310425 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_303504fb62ac468c8985d6d1b3f25851" } modify_external_catalogs: true
I20250625 07:52:34.310786 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f8acc41371a4800a82de306adce74db" } modify_external_catalogs: true
I20250625 07:52:34.310997 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f8acc41371a4800a82de306adce74db" } modify_external_catalogs: true
I20250625 07:52:34.311072 1163 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.311249 1163 ts_tablet_manager.cc:1397] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.004s
I20250625 07:52:34.312544 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.314808 1158 tablet_bootstrap.cc:654] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:34.318604 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f771628379842989b9b2099de65e2a0 consensus state has no leader
I20250625 07:52:34.319221 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b47065e0382b4b9a936613a0ad84695d
I20250625 07:52:34.320073 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1a536939b2ae4f5b9dc8b8bc27da686f" } new_table_name: "default.Table_cd7eb16c71bb49cba917660f5a1c1b71" modify_external_catalogs: true
I20250625 07:52:34.321280 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e046fc4755bc40769ba73e88edfe6b53" } new_table_name: "default.Table_a963a1a41d5a4ac8bc0797be58a18631" modify_external_catalogs: true
I20250625 07:52:34.323134 879 tablet_service.cc:1515] Processing DeleteTablet for tablet b47065e0382b4b9a936613a0ad84695d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.319221 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_859d1eab20d84c2ea2ca2af35baab1da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.327750 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_859d1eab20d84c2ea2ca2af35baab1da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.328996 597 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6bfd3c9a62a2424cbcbb5c7f812b1027
I20250625 07:52:34.329346 597 master_service.cc:948] ReplaceTablet: received request to replace tablet b620bb670cd94dce999cbe1dabc6125a from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:34.330091 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 168f1a13533045d89ba1f07a72d4ff63
I20250625 07:52:34.329986 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_35af5e37776542d28c4089157dfe9b57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.330121 1163 raft_consensus.cc:357] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.330291 1163 raft_consensus.cc:383] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.330348 1163 raft_consensus.cc:738] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.322656 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b47065e0382b4b9a936613a0ad84695d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.324471 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f9a94dc1432d4cbdbf3eecacb1429c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.330740 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a0e57062ab164088b1a7cf52ffe4dd18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.331195 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0e57062ab164088b1a7cf52ffe4dd18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.331511 1163 consensus_queue.cc:260] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.332017 1163 ts_tablet_manager.cc:1428] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
W20250625 07:52:34.330740 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35af5e37776542d28c4089157dfe9b57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.336412 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.336548 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.336655 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 366d3e6fcc9d49be8fbd2cd070632861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.336994 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 541c983d64f348539979f74199f79e59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.337564 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ec29e4294864fbabdff5a04dcf5b163 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.339136 1163 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.340572 1163 tablet_bootstrap.cc:654] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.342698 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 4da889781dbd41aebac86e9833cf2b7e (table default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba]). Expected version 1 got 0
I20250625 07:52:34.344174 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_65c91dd95e0d4b24854320d405dfdbfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.348203 879 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.349315 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c0bbd21d4aa41b2b92473bffe7dc05a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.353040 2178 raft_consensus.cc:491] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.353185 2178 raft_consensus.cc:513] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.353765 2178 leader_election.cc:290] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:34.354624 985 leader_election.cc:343] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: fcc3afc210a540058294121d2b83c66d
W20250625 07:52:34.354768 986 leader_election.cc:343] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: fcc3afc210a540058294121d2b83c66d
I20250625 07:52:34.354838 986 leader_election.cc:304] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.355151 2178 raft_consensus.cc:2747] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.192540 1162 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.222857 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.276371 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.357332 879 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.197759 2161 raft_consensus.cc:2747] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.357887 851 leader_election.cc:343] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.358016 851 leader_election.cc:304] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:34.358350 851 leader_election.cc:343] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.358454 851 leader_election.cc:304] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.358570 1162 tablet_bootstrap.cc:654] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.358889 2112 raft_consensus.cc:2747] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.359040 2112 raft_consensus.cc:2747] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.192467 713 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.359575 713 leader_election.cc:304] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.359799 2161 raft_consensus.cc:2747] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.359874 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 167462us
I20250625 07:52:34.360394 1162 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.361317 1162 ts_tablet_manager.cc:1397] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.169s user 0.002s sys 0.002s
I20250625 07:52:34.362176 1162 raft_consensus.cc:357] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.363538 1162 raft_consensus.cc:383] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.363723 1162 raft_consensus.cc:738] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.364110 1162 consensus_queue.cc:260] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.365105 1162 ts_tablet_manager.cc:1428] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 07:52:34.366596 2161 raft_consensus.cc:491] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.366708 2161 raft_consensus.cc:513] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.367188 2161 leader_election.cc:290] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.367496 1162 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.369050 1162 tablet_bootstrap.cc:654] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.345989 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_33cd2aff4e6b46b7a1aa026c219cd20b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.371815 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33cd2aff4e6b46b7a1aa026c219cd20b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.348326 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bff347d838fe4e56a195a957d0deb742
I20250625 07:52:34.374424 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bff347d838fe4e56a195a957d0deb742 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:34.349733 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 consensus state has no leader
W20250625 07:52:34.344769 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c91dd95e0d4b24854320d405dfdbfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.371186 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bc387934e5204a3e9ce6f13d1dc3f777
I20250625 07:52:34.379071 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.379729 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.381417 1158 tablet_bootstrap.cc:492] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.381562 1158 ts_tablet_manager.cc:1397] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20250625 07:52:34.382297 1158 raft_consensus.cc:357] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.382481 1158 raft_consensus.cc:383] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.382551 1158 raft_consensus.cc:738] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.382997 1158 consensus_queue.cc:260] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.389627 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet fc6c3be49be14ecbb3c26ef107ede015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.390220 2161 raft_consensus.cc:491] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.390375 2161 raft_consensus.cc:513] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.390918 2161 leader_election.cc:290] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.391790 1158 ts_tablet_manager.cc:1428] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250625 07:52:34.393718 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.393879 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.393945 756 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.394287 1166 tablet_replica.cc:331] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.394483 1166 raft_consensus.cc:2241] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.394618 1166 raft_consensus.cc:2270] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.395133 1166 ts_tablet_manager.cc:1905] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.395255 746 tablet_service.cc:1515] Processing DeleteTablet for tablet b47065e0382b4b9a936613a0ad84695d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.395553 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 168f1a13533045d89ba1f07a72d4ff63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.396104 756 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.396772 752 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.393906 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.397434 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.397606 2286 raft_consensus.cc:491] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.397738 2286 raft_consensus.cc:513] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.398547 2286 leader_election.cc:290] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.399106 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.399132 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:34.399580 850 leader_election.cc:343] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.399806 851 leader_election.cc:343] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.399907 851 leader_election.cc:304] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.400478 2286 raft_consensus.cc:2747] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.402007 1158 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.402715 1166 ts_tablet_manager.cc:1918] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.402904 1166 log.cc:1199] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:34.403272 1166 ts_tablet_manager.cc:1939] T 0bfcef152c9844a59989c532bd60c07d P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.403710 1158 tablet_bootstrap.cc:654] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.404412 1166 tablet_replica.cc:331] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.404639 1166 raft_consensus.cc:2241] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:34.404897 1166 pending_rounds.cc:70] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:34.404979 1166 pending_rounds.cc:77] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432259876814848 op_type: NO_OP noop_request { }
I20250625 07:52:34.405234 1166 raft_consensus.cc:2887] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:34.405370 1166 raft_consensus.cc:2270] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.407325 1166 ts_tablet_manager.cc:1905] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.413895 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet bff347d838fe4e56a195a957d0deb742 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.417891 1163 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.418176 1163 ts_tablet_manager.cc:1397] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.006s
I20250625 07:52:34.419250 1163 raft_consensus.cc:357] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.419585 1163 raft_consensus.cc:383] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.419715 1163 raft_consensus.cc:738] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.420040 1163 consensus_queue.cc:260] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.423955 1163 ts_tablet_manager.cc:1428] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:34.424405 1163 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.429214 754 tablet_service.cc:1515] Processing DeleteTablet for tablet bff347d838fe4e56a195a957d0deb742 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.430066 758 tablet_service.cc:1515] Processing DeleteTablet for tablet bc387934e5204a3e9ce6f13d1dc3f777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.432097 2178 raft_consensus.cc:491] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.432233 2178 raft_consensus.cc:513] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.432878 2178 leader_election.cc:290] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:34.433945 985 leader_election.cc:343] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f9a94dc1432d4cbdbf3eecacb1429c23
W20250625 07:52:34.434245 986 leader_election.cc:343] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:34.434319 986 leader_election.cc:304] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.434736 2178 raft_consensus.cc:2747] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.435101 1166 ts_tablet_manager.cc:1918] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:34.438486 1166 log.cc:1199] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:34.439221 1166 ts_tablet_manager.cc:1939] T cf175e17865e40ceb26aa955b91c2c13 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.441336 1166 tablet_replica.cc:331] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.445982 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1405e55703324bc7a1e26fe2c349bda7 (table default.Table_cd7eb16c71bb49cba917660f5a1c1b71 [id=bde8f673978c457e8a36b1173b315a06]). Expected version 1 got 0
I20250625 07:52:34.447530 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f04016cda7f746d49e30be3b1c9c18b6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.450326 1166 raft_consensus.cc:2241] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.451146 1166 raft_consensus.cc:2270] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.452241 1166 ts_tablet_manager.cc:1905] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.459714 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.465510 2286 raft_consensus.cc:491] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.465692 2286 raft_consensus.cc:513] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.466226 2286 leader_election.cc:290] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.466871 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.467000 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.467298 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:34.467497 911 raft_consensus.cc:2466] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:34.467506 851 leader_election.cc:343] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.468066 850 leader_election.cc:343] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.468201 850 leader_election.cc:304] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.468648 2286 raft_consensus.cc:2747] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.469236 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 34a338005a7b44b9b0fba7a7096cfcb8 because the tablet was not found. No further retry: Not found: Tablet not found: 34a338005a7b44b9b0fba7a7096cfcb8
I20250625 07:52:34.469513 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0bfcef152c9844a59989c532bd60c07d (table default.Table_1f8502fb577748268f87ac63e15ec4f5 [id=b6cfc505ed734958966b9e75b0e39415]) successfully deleted
I20250625 07:52:34.471508 1163 tablet_bootstrap.cc:654] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.472487 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1405e55703324bc7a1e26fe2c349bda7 (table default.Table_cd7eb16c71bb49cba917660f5a1c1b71 [id=bde8f673978c457e8a36b1173b315a06]). Expected version 1 got 0
I20250625 07:52:34.472877 592 catalog_manager.cc:5582] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:34.473312 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:34.473783 1158 tablet_bootstrap.cc:492] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.473920 1158 ts_tablet_manager.cc:1397] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.033s
I20250625 07:52:34.474705 1158 raft_consensus.cc:357] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.474864 1158 raft_consensus.cc:383] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.474915 1158 raft_consensus.cc:738] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.475124 1158 consensus_queue.cc:260] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.479395 1166 ts_tablet_manager.cc:1918] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.479740 1161 ts_tablet_manager.cc:1918] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.479890 1161 log.cc:1199] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0f566cc9cdd447798a5cb408ef9cd35a
I20250625 07:52:34.480244 1161 ts_tablet_manager.cc:1939] T 0f566cc9cdd447798a5cb408ef9cd35a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:34.481519 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0f566cc9cdd447798a5cb408ef9cd35a (table default.Table_27c3e9984caf46e0ac463d7dc5b7edc4 [id=83968a0333c94ece83f709b30b555928]) successfully deleted
I20250625 07:52:34.484447 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eb542ac24f6405687c1067f2885bec6. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:34.484750 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [sys.catalog]: This master's current role is: FOLLOWER
I20250625 07:52:34.485044 1166 log.cc:1199] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/f04016cda7f746d49e30be3b1c9c18b6
I20250625 07:52:34.485363 1166 ts_tablet_manager.cc:1939] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.485373 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet ba0735d576f947d395bee8d1b865f2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.485500 2235 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:34.485847 1158 ts_tablet_manager.cc:1428] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:34.485721 2235 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:34.486176 1166 tablet_replica.cc:331] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.486356 1166 raft_consensus.cc:2241] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.486387 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:34.486516 1166 raft_consensus.cc:2270] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.487003 1166 ts_tablet_manager.cc:1905] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.488289 713 leader_election.cc:343] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.488446 713 leader_election.cc:343] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.488986 2235 leader_election.cc:290] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317), 3eb542ac24f6405687c1067f2885bec6 (127.0.104.60:33457)
I20250625 07:52:34.492383 1158 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.497387 1163 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.497589 1163 ts_tablet_manager.cc:1397] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250625 07:52:34.498365 1163 raft_consensus.cc:357] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.498593 1163 raft_consensus.cc:383] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.498687 1163 raft_consensus.cc:738] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.498929 1163 consensus_queue.cc:260] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.499508 1163 ts_tablet_manager.cc:1428] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.502269 1158 tablet_bootstrap.cc:654] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.503952 1096 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:34.505842 1908 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51466 (request call id 0) took 5020 ms. Trace:
I20250625 07:52:34.505899 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5b7b76e7ccd144bd913b3908d3aac577"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.506489 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b7b76e7ccd144bd913b3908d3aac577 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.509147 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4e9b498709a94e19a3f242fad2d3abb2" } new_table_name: "default.Table_250afd2ed99c43b4a24721d178e5e372" modify_external_catalogs: true
I20250625 07:52:34.507921 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.515174 1163 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.516852 1163 tablet_bootstrap.cc:654] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.525993 1163 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.529358 1163 ts_tablet_manager.cc:1397] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20250625 07:52:34.531319 1163 raft_consensus.cc:357] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.534297 1163 raft_consensus.cc:383] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.534524 1163 raft_consensus.cc:738] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.534873 1163 consensus_queue.cc:260] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.538362 1163 ts_tablet_manager.cc:1428] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250625 07:52:34.539714 1163 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:34.545043 713 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 56221.2us
I20250625 07:52:34.549300 2045 consensus_queue.cc:1035] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:34.545178 1163 tablet_bootstrap.cc:654] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.546442 1160 ts_tablet_manager.cc:1918] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.552160 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.553254 1160 log.cc:1199] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:34.553637 1160 ts_tablet_manager.cc:1939] T 0bfcef152c9844a59989c532bd60c07d P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.554324 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1145) took 2200 ms. Trace:
I20250625 07:52:34.554461 1160 rpcz_store.cc:276] 0625 07:52:32.353853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.355460 (+ 1607us) service_pool.cc:225] Handling call
0625 07:52:34.103075 (+1747615us) ts_tablet_manager.cc:1145] Deleting tablet 0bfcef152c9844a59989c532bd60c07d
0625 07:52:34.103086 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.351627 (+248541us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.423506 (+ 71879us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.554307 (+130801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1747416}
I20250625 07:52:34.554975 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0bfcef152c9844a59989c532bd60c07d (table default.Table_1f8502fb577748268f87ac63e15ec4f5 [id=b6cfc505ed734958966b9e75b0e39415]) successfully deleted
I20250625 07:52:34.555393 1160 tablet_replica.cc:331] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.555605 1160 raft_consensus.cc:2241] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.555773 1160 raft_consensus.cc:2270] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.556269 1160 ts_tablet_manager.cc:1905] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.557361 713 leader_election.cc:343] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f04016cda7f746d49e30be3b1c9c18b6
I20250625 07:52:34.557492 713 leader_election.cc:304] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:34.557677 713 raft_consensus.cc:2684] T f04016cda7f746d49e30be3b1c9c18b6 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:34.558765 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "3eb542ac24f6405687c1067f2885bec6" is_pre_election: true
I20250625 07:52:34.562007 1163 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.567972 2286 raft_consensus.cc:491] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.568835 1163 ts_tablet_manager.cc:1397] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20250625 07:52:34.568779 2286 raft_consensus.cc:513] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.569437 2286 leader_election.cc:290] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.569890 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3d8b6b5c06248879e6052f6ae7fe46e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.569905 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3d8b6b5c06248879e6052f6ae7fe46e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:34.571406 850 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.571588 851 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.571707 851 leader_election.cc:304] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.571771 1886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "429026e8d98d46339db143f84897a092" instance_seqno: 1750837930718830) as {username='slave'} at 127.0.104.2:51151; Asking this server to re-register.
I20250625 07:52:34.571925 2286 raft_consensus.cc:2747] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.572659 1163 raft_consensus.cc:357] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.574047 1163 raft_consensus.cc:383] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.574700 1163 raft_consensus.cc:738] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.575176 961 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:34.575433 1163 consensus_queue.cc:260] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.558228 713 leader_election.cc:304] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.575801 2045 raft_consensus.cc:2802] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:34.575886 2045 raft_consensus.cc:491] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:34.575949 2045 raft_consensus.cc:3058] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.576223 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet cf175e17865e40ceb26aa955b91c2c13 (table default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]) successfully deleted
I20250625 07:52:34.576642 1163 ts_tablet_manager.cc:1428] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250625 07:52:34.577096 1158 tablet_bootstrap.cc:492] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.577239 1158 ts_tablet_manager.cc:1397] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.008s
I20250625 07:52:34.577950 1158 raft_consensus.cc:357] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.578100 1158 raft_consensus.cc:383] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.578153 1158 raft_consensus.cc:738] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.578348 1158 consensus_queue.cc:260] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.578814 1158 ts_tablet_manager.cc:1428] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:34.578923 1162 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.579033 1162 ts_tablet_manager.cc:1397] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.019s
I20250625 07:52:34.579087 1158 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.578994 2045 raft_consensus.cc:513] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.579556 2045 leader_election.cc:290] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.579902 1162 raft_consensus.cc:357] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.580037 1162 raft_consensus.cc:383] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.580091 1162 raft_consensus.cc:738] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.580262 1162 consensus_queue.cc:260] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.581525 1162 ts_tablet_manager.cc:1428] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:34.581787 1162 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.583161 1162 tablet_bootstrap.cc:654] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.586410 1161 tablet_replica.cc:331] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:34.586575 1161 raft_consensus.cc:2241] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.586700 1161 raft_consensus.cc:2270] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.587100 1166 ts_tablet_manager.cc:1918] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.587186 1166 log.cc:1199] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/f4bb74391a1c4e2792226d259a07f977
I20250625 07:52:34.587178 1161 ts_tablet_manager.cc:1905] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.587488 1166 ts_tablet_manager.cc:1939] T f4bb74391a1c4e2792226d259a07f977 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.568082 1160 ts_tablet_manager.cc:1918] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.587767 1160 log.cc:1199] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:34.588068 1160 ts_tablet_manager.cc:1939] T cf175e17865e40ceb26aa955b91c2c13 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.590039 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 2771159858fc41a1ad930229986edd92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.590209 2178 raft_consensus.cc:491] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.590312 2178 raft_consensus.cc:513] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.590876 2178 leader_election.cc:290] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:34.591001 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b47065e0382b4b9a936613a0ad84695d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b47065e0382b4b9a936613a0ad84695d already in progress: creating tablet
W20250625 07:52:34.591349 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 168f1a13533045d89ba1f07a72d4ff63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168f1a13533045d89ba1f07a72d4ff63 already in progress: creating tablet
I20250625 07:52:34.591735 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet f04016cda7f746d49e30be3b1c9c18b6 (table default.Table_634beccfc1a34257bd0bc04d1d020530 [id=989d9b62801c467e8bba2e03c0f13395]) successfully deleted
I20250625 07:52:34.596410 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.596531 746 tablet_service.cc:1515] Processing DeleteTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b with delete_type TABLET_DATA_DELETED (replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.597229 746 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.597891 2286 raft_consensus.cc:491] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.598021 2286 raft_consensus.cc:513] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.598277 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:34.598588 2286 leader_election.cc:290] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:34.599164 986 leader_election.cc:343] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:34.599848 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:34.600593 851 leader_election.cc:343] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.601011 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:34.601397 850 leader_election.cc:343] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.601497 850 leader_election.cc:304] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.601949 1166 tablet_replica.cc:331] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.602660 2286 raft_consensus.cc:2747] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.607585 1886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" instance_seqno: 1750837931198880) as {username='slave'} at 127.0.104.3:43781; Asking this server to re-register.
I20250625 07:52:34.608765 1096 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:34.608749 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5199775108e44a9780009e4788a861ad" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:34.608847 1163 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.608894 907 raft_consensus.cc:3058] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.610819 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1146) took 2154 ms. Trace:
W20250625 07:52:34.610968 985 leader_election.cc:343] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:34.610899 1160 rpcz_store.cc:276] 0625 07:52:32.456632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.456708 (+ 76us) service_pool.cc:225] Handling call
0625 07:52:34.555274 (+2098566us) ts_tablet_manager.cc:1145] Deleting tablet cf175e17865e40ceb26aa955b91c2c13
0625 07:52:34.555350 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.564921 (+ 9571us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.567989 (+ 3068us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.610272 (+ 42283us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x61800000d518. stack: 00007f8e6ebed336 00007f8e6ebed0f0 00007f8e6dbe33ad 0000000000971004 0000000000970fcd 000000000097099a 00007f8e6ebd5c3d 00007f8e6ebd6cae 00007f8e8226b80d 00007f8e821ed0f3 00007f8e7e43c5a3 00007f8e7e46e1c4 00007f8e7e44fc20 00007f8e7e44f9ad 0000000000a7573c 00007f8e6ec329df
0625 07:52:34.610801 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35745536,"tablet-delete.queue_time_us":2098383}
I20250625 07:52:34.611063 985 leader_election.cc:304] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.611200 1160 tablet_replica.cc:331] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.611384 1160 raft_consensus.cc:2241] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.611438 2178 raft_consensus.cc:2747] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.611526 1160 raft_consensus.cc:2270] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.611649 907 raft_consensus.cc:2466] T 5199775108e44a9780009e4788a861ad P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:34.612246 1160 ts_tablet_manager.cc:1905] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.612637 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet cf175e17865e40ceb26aa955b91c2c13 (table default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]) successfully deleted
I20250625 07:52:34.614802 1885 ts_manager.cc:194] Registered new tserver with Master: 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.618712 1158 tablet_bootstrap.cc:654] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.622357 1885 ts_manager.cc:194] Registered new tserver with Master: 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.626284 2178 raft_consensus.cc:491] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.626417 2178 raft_consensus.cc:513] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.627020 2178 leader_election.cc:290] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:34.628093 985 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4c0bbd21d4aa41b2b92473bffe7dc05a
W20250625 07:52:34.628818 986 leader_election.cc:343] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:34.628921 986 leader_election.cc:304] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.629305 2178 raft_consensus.cc:2747] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.631668 1160 ts_tablet_manager.cc:1918] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.631811 1160 log.cc:1199] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/fdad7dc1b27e4d87a747642a6d47230c
I20250625 07:52:34.632098 1160 ts_tablet_manager.cc:1939] T fdad7dc1b27e4d87a747642a6d47230c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.632876 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1147) took 2175 ms. Trace:
I20250625 07:52:34.632970 1160 rpcz_store.cc:276] 0625 07:52:32.457238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.457287 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:34.611159 (+2153872us) ts_tablet_manager.cc:1145] Deleting tablet fdad7dc1b27e4d87a747642a6d47230c
0625 07:52:34.611168 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.617948 (+ 6780us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.631557 (+ 13609us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.632855 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":213,"tablet-delete.queue_time_us":2153723}
I20250625 07:52:34.633440 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet fdad7dc1b27e4d87a747642a6d47230c (table default.Table_ba10e2f968a543d4a5173431533d7e72 [id=6da9e5a92547450b83830b27727c69ef]) successfully deleted
I20250625 07:52:34.634075 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1148) took 2013 ms. Trace:
I20250625 07:52:34.634155 1160 rpcz_store.cc:276] 0625 07:52:32.620965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.622491 (+ 1526us) service_pool.cc:225] Handling call
0625 07:52:34.633961 (+2011470us) ts_tablet_manager.cc:1145] Deleting tablet f2ee3980ed9747af87329fd3a1dfd699
0625 07:52:34.633971 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.634060 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2011143}
W20250625 07:52:34.634541 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f2ee3980ed9747af87329fd3a1dfd699 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ee3980ed9747af87329fd3a1dfd699 already in progress: creating tablet
I20250625 07:52:34.634877 1160 tablet_replica.cc:331] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.635150 1160 raft_consensus.cc:2241] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.635339 1160 raft_consensus.cc:2270] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.635299 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8555014fea674cb9873c91ccce23dee0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.635581 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b649eef37b514286a98a4bc732c1870a" } modify_external_catalogs: true
I20250625 07:52:34.635739 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b649eef37b514286a98a4bc732c1870a" } modify_external_catalogs: true
W20250625 07:52:34.635883 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8555014fea674cb9873c91ccce23dee0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.636196 1160 ts_tablet_manager.cc:1905] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.636296 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_99eaf4f7bbec455ab0c9291fedfa221d" } new_table_name: "default.Table_ec11ec9d97c040abb492fec0547ba805" modify_external_catalogs: true
I20250625 07:52:34.637096 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_346480a0ddc54315b4df65b09d71294f" } new_table_name: "default.Table_7c6816b996324ddab0fc59063a2191b7" modify_external_catalogs: true
I20250625 07:52:34.637530 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c15ba9c8f684ea4bc03053e7370eefe" } modify_external_catalogs: true
I20250625 07:52:34.637686 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c15ba9c8f684ea4bc03053e7370eefe" } modify_external_catalogs: true
I20250625 07:52:34.638804 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_364620262bdb4f57bb3be61e6f9aa395" } modify_external_catalogs: true
I20250625 07:52:34.639004 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_364620262bdb4f57bb3be61e6f9aa395" } modify_external_catalogs: true
I20250625 07:52:34.642351 1160 ts_tablet_manager.cc:1918] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.642452 1160 log.cc:1199] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:34.642727 1160 ts_tablet_manager.cc:1939] T 0657163b9a0d4f7f99f57f7fd1f32462 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.643373 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1149) took 1955 ms. Trace:
I20250625 07:52:34.643458 1160 rpcz_store.cc:276] 0625 07:52:32.687742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.691924 (+ 4182us) service_pool.cc:225] Handling call
0625 07:52:34.634836 (+1942912us) ts_tablet_manager.cc:1145] Deleting tablet 0657163b9a0d4f7f99f57f7fd1f32462
0625 07:52:34.634844 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.639203 (+ 4359us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.642263 (+ 3060us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.643354 (+ 1091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942568}
I20250625 07:52:34.643700 2286 raft_consensus.cc:491] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.643802 2286 raft_consensus.cc:513] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.643899 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e1f71864f5a42d9a26edb7293a99df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.644374 2286 leader_election.cc:290] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:34.644388 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e1f71864f5a42d9a26edb7293a99df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.645089 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.645547 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0657163b9a0d4f7f99f57f7fd1f32462 (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
W20250625 07:52:34.646085 850 leader_election.cc:343] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.646384 1160 tablet_replica.cc:331] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.646564 1160 raft_consensus.cc:2241] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.646728 1160 raft_consensus.cc:2270] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.647218 1160 ts_tablet_manager.cc:1905] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.646013 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6c61bacc767d4162844a114ff368d970"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.647809 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c61bacc767d4162844a114ff368d970 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:34.651314 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b13c1f3f0e04ea3a1b5077d303e7409 consensus state has no leader
I20250625 07:52:34.651731 588 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8cb3fc4d630545a2aba37194c9a96384
I20250625 07:52:34.652081 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2a0303308113473b8a8d3f047b48e2db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.652520 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a0303308113473b8a8d3f047b48e2db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.652837 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_976b8466313948adb4c34681120d4eac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.653302 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e4425d3b61cf47d58737fcf7a4dcac7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.653730 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4425d3b61cf47d58737fcf7a4dcac7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:34.653276 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_976b8466313948adb4c34681120d4eac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.654183 1160 ts_tablet_manager.cc:1918] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.654284 1160 log.cc:1199] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c263b671be384cd097041c60306034de
I20250625 07:52:34.654590 1160 ts_tablet_manager.cc:1939] T c263b671be384cd097041c60306034de P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.654947 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 768c9551d33346718af2670a01a626b0
I20250625 07:52:34.655614 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 768c9551d33346718af2670a01a626b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.655649 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cfb63fdad31e4cada3c0e785a501be1d" } modify_external_catalogs: true
I20250625 07:52:34.655819 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cfb63fdad31e4cada3c0e785a501be1d" } modify_external_catalogs: true
I20250625 07:52:34.656157 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 768c9551d33346718af2670a01a626b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.656219 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1150) took 1968 ms. Trace:
I20250625 07:52:34.656273 1160 rpcz_store.cc:276] 0625 07:52:32.688171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.692321 (+ 4150us) service_pool.cc:225] Handling call
0625 07:52:34.646339 (+1954018us) ts_tablet_manager.cc:1145] Deleting tablet c263b671be384cd097041c60306034de
0625 07:52:34.646348 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.650555 (+ 4207us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.654103 (+ 3548us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.656206 (+ 2103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953866}
I20250625 07:52:34.656744 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c263b671be384cd097041c60306034de (table default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]) successfully deleted
I20250625 07:52:34.657110 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1151) took 1947 ms. Trace:
I20250625 07:52:34.657195 1160 rpcz_store.cc:276] 0625 07:52:32.709982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.710995 (+ 1013us) service_pool.cc:225] Handling call
0625 07:52:34.656985 (+1945990us) ts_tablet_manager.cc:1145] Deleting tablet 8a899790713d4646a850971bc3aebd9f
0625 07:52:34.656995 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.657094 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945800}
W20250625 07:52:34.657615 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8a899790713d4646a850971bc3aebd9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a899790713d4646a850971bc3aebd9f already in progress: creating tablet
I20250625 07:52:34.657980 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1152) took 1923 ms. Trace:
I20250625 07:52:34.658097 1160 rpcz_store.cc:276] 0625 07:52:32.734594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:32.734764 (+ 170us) service_pool.cc:225] Handling call
0625 07:52:34.657879 (+1923115us) ts_tablet_manager.cc:1145] Deleting tablet 8d34bfe377ba421daede5ae68848a2d4
0625 07:52:34.657888 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.657965 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861188}
W20250625 07:52:34.658535 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8d34bfe377ba421daede5ae68848a2d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d34bfe377ba421daede5ae68848a2d4 already in progress: creating tablet
I20250625 07:52:34.658888 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1163) took 1529 ms. Trace:
I20250625 07:52:34.658999 1160 rpcz_store.cc:276] 0625 07:52:33.129177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.130570 (+ 1393us) service_pool.cc:225] Handling call
0625 07:52:34.658791 (+1528221us) ts_tablet_manager.cc:1145] Deleting tablet 33441054063e47deaba2ad04ae6ae764
0625 07:52:34.658800 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.658872 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527967}
W20250625 07:52:34.659520 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 33441054063e47deaba2ad04ae6ae764 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33441054063e47deaba2ad04ae6ae764 already in progress: creating tablet
I20250625 07:52:34.659878 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1166) took 1473 ms. Trace:
I20250625 07:52:34.659994 1160 rpcz_store.cc:276] 0625 07:52:33.186263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.202911 (+ 16648us) service_pool.cc:225] Handling call
0625 07:52:34.659785 (+1456874us) ts_tablet_manager.cc:1145] Deleting tablet 74fa6e5ac74444729645a27fc2945bc6
0625 07:52:34.659794 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.659862 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456656}
W20250625 07:52:34.660441 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 74fa6e5ac74444729645a27fc2945bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74fa6e5ac74444729645a27fc2945bc6 already in progress: creating tablet
I20250625 07:52:34.660810 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1168) took 1449 ms. Trace:
I20250625 07:52:34.661131 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d3b790bb56d44a24b68373d465cf0a3b" } new_table_name: "default.Table_903b3fa5b06348ec8f2b92c77da28e7b" modify_external_catalogs: true
I20250625 07:52:34.660923 1160 rpcz_store.cc:276] 0625 07:52:33.211439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.214545 (+ 3106us) service_pool.cc:225] Handling call
0625 07:52:34.660714 (+1446169us) ts_tablet_manager.cc:1145] Deleting tablet b8b5bb3fe3ef4bfdba93b862784a9f61
0625 07:52:34.660723 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.660795 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445935}
I20250625 07:52:34.663532 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1169) took 1430 ms. Trace:
I20250625 07:52:34.663654 1160 rpcz_store.cc:276] 0625 07:52:33.232859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.232910 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:34.663384 (+1430474us) ts_tablet_manager.cc:1145] Deleting tablet a2481ca7e8db42a3a51270158bb2e2fa
0625 07:52:34.663393 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.663515 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1430263}
W20250625 07:52:34.664090 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a2481ca7e8db42a3a51270158bb2e2fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2481ca7e8db42a3a51270158bb2e2fa already in progress: creating tablet
I20250625 07:52:34.664394 1160 tablet_replica.cc:331] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.664594 1160 raft_consensus.cc:2241] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.664669 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_77a9000fc7304886beb0d06b143debdc" } modify_external_catalogs: true
I20250625 07:52:34.664783 1160 raft_consensus.cc:2270] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.664935 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_77a9000fc7304886beb0d06b143debdc" } modify_external_catalogs: true
I20250625 07:52:34.665259 1160 ts_tablet_manager.cc:1905] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.667443 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 60edc8fa184542a0b54b89736831730d (table default.Table_cb409c5e3a884ffaac6321d86f5b0e03 [id=f28fa7e6c39c478c9b313cb0b0963fd5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 60edc8fa184542a0b54b89736831730d consensus state has no leader
I20250625 07:52:34.670197 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8d97e47e913f4458bb17f7c95671f879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.670784 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d97e47e913f4458bb17f7c95671f879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.672250 1160 ts_tablet_manager.cc:1918] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.672411 1160 log.cc:1199] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:34.672789 1160 ts_tablet_manager.cc:1939] T 0b7daf759e324c1db853fdbaab76c721 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.672819 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 1898ff5b7626478bb997f7a433477498 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:34.673575 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1170) took 1423 ms. Trace:
I20250625 07:52:34.673708 1160 rpcz_store.cc:276] 0625 07:52:33.250032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.250082 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:34.664340 (+1414258us) ts_tablet_manager.cc:1145] Deleting tablet 0b7daf759e324c1db853fdbaab76c721
0625 07:52:34.664349 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.668653 (+ 4304us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.672165 (+ 3512us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:34.673558 (+ 1393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414098}
I20250625 07:52:34.674113 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_42c59eeeff4e404d8aa4c201145576b8" } new_table_name: "default.Table_69c013cd0c6a4f5397f00282c3269577" modify_external_catalogs: true
I20250625 07:52:34.674181 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0b7daf759e324c1db853fdbaab76c721 (table default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]) successfully deleted
I20250625 07:52:34.674453 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1171) took 1403 ms. Trace:
I20250625 07:52:34.674530 1160 rpcz_store.cc:276] 0625 07:52:33.271087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.271129 (+ 42us) service_pool.cc:225] Handling call
0625 07:52:34.674333 (+1403204us) ts_tablet_manager.cc:1145] Deleting tablet f35ac535578d4681ac998a530b1f1d4f
0625 07:52:34.674342 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.674437 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402989}
W20250625 07:52:34.674916 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f35ac535578d4681ac998a530b1f1d4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f35ac535578d4681ac998a530b1f1d4f already in progress: creating tablet
I20250625 07:52:34.675277 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1172) took 1363 ms. Trace:
I20250625 07:52:34.675398 1160 rpcz_store.cc:276] 0625 07:52:33.311463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:33.311519 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:34.675183 (+1363664us) ts_tablet_manager.cc:1145] Deleting tablet 68f903943e9a4151b6fa1e71d1c05473
0625 07:52:34.675192 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:34.675262 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363482}
I20250625 07:52:34.675796 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f16cca6477ee43538ba95d01134aa5a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.676043 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 68f903943e9a4151b6fa1e71d1c05473 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f903943e9a4151b6fa1e71d1c05473 already in progress: creating tablet
W20250625 07:52:34.676299 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f16cca6477ee43538ba95d01134aa5a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.676270 1166 raft_consensus.cc:2241] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.645089 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.676473 1166 raft_consensus.cc:2270] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:34.676945 851 leader_election.cc:343] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.677021 1166 ts_tablet_manager.cc:1905] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.677047 851 leader_election.cc:304] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.677505 2286 raft_consensus.cc:2747] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.678157 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 034903d699584feb91f35cbb2f62701a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 034903d699584feb91f35cbb2f62701a already in progress: creating tablet
I20250625 07:52:34.678906 1162 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:34.679489 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 24c767b82ed24df1b145ff2df33ba155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c767b82ed24df1b145ff2df33ba155 already in progress: creating tablet
W20250625 07:52:34.680071 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 953cfe2c224043eca4da735c2a5dbebc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 953cfe2c224043eca4da735c2a5dbebc already in progress: creating tablet
W20250625 07:52:34.680645 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7d42f5f3910742f0942742eb77ea1435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d42f5f3910742f0942742eb77ea1435 already in progress: creating tablet
I20250625 07:52:34.680934 1162 ts_tablet_manager.cc:1397] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.002s
W20250625 07:52:34.681417 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a7dd3b2533244b93a88d12660ea0a4b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7dd3b2533244b93a88d12660ea0a4b9 already in progress: creating tablet
I20250625 07:52:34.682257 1162 raft_consensus.cc:357] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.682615 1160 tablet_replica.cc:331] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.682819 1160 raft_consensus.cc:2241] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.682991 1160 raft_consensus.cc:2270] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.683466 1160 ts_tablet_manager.cc:1905] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.683583 1162 raft_consensus.cc:383] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.683678 1162 raft_consensus.cc:738] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.683936 1162 consensus_queue.cc:260] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.684105 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6b33905fc1b444d6876011512196d67b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.684585 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b33905fc1b444d6876011512196d67b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.686740 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d4d367cc123e4bbda8054d1d61d29be1
I20250625 07:52:34.688952 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b74d8f0eabf1410fadaadb98344b3f62
I20250625 07:52:34.689322 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.689425 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.690338 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:34.690443 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dde275b2d99a4b0cb008a2fba9945960 has no consensus state
I20250625 07:52:34.690845 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.691692 1162 ts_tablet_manager.cc:1428] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20250625 07:52:34.697247 1162 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.698388 1166 ts_tablet_manager.cc:1918] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.698552 1166 log.cc:1199] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/110fe299c7c44df69f459da72a1138a5
I20250625 07:52:34.698892 1166 ts_tablet_manager.cc:1939] T 110fe299c7c44df69f459da72a1138a5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.702733 1160 ts_tablet_manager.cc:1918] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.702834 1160 log.cc:1199] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0aeb7077591f466eb77e3096f6b1d74b
I20250625 07:52:34.703105 1160 ts_tablet_manager.cc:1939] T 0aeb7077591f466eb77e3096f6b1d74b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.704035 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0aeb7077591f466eb77e3096f6b1d74b (table default.Table_1225d003be2b4d048266736844a78a01 [id=357521ce57d54dafae9cfd2efbe731c2]) successfully deleted
W20250625 07:52:34.708974 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3823279d7e06435484f1b82836ebcf39 consensus state has no leader
I20250625 07:52:34.712560 1162 tablet_bootstrap.cc:654] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.712998 1166 tablet_replica.cc:331] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.713248 1166 raft_consensus.cc:2241] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.713493 1166 raft_consensus.cc:2270] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.713523 1158 tablet_bootstrap.cc:492] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.713655 1158 ts_tablet_manager.cc:1397] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.135s user 0.005s sys 0.002s
I20250625 07:52:34.714109 1166 ts_tablet_manager.cc:1905] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.714309 1158 raft_consensus.cc:357] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.714468 1158 raft_consensus.cc:383] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.714519 1158 raft_consensus.cc:738] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.714766 1158 consensus_queue.cc:260] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.715183 1158 ts_tablet_manager.cc:1428] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:34.715402 1158 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.716933 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3529a591e1ea4cc5803cc8701e60808a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.717453 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3529a591e1ea4cc5803cc8701e60808a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.719841 1158 tablet_bootstrap.cc:654] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.722824 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 1898ff5b7626478bb997f7a433477498 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a29a9ba27ab4eb494e1b91e4b33aa75) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.724190 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1199 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:34.725474 1821 leader_election.cc:304] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a; no voters: 3eb542ac24f6405687c1067f2885bec6, 789c06e7de2842a1b40e97e0778ab6c9
I20250625 07:52:34.725883 2235 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 07:52:34.727535 1163 tablet_bootstrap.cc:654] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:34.728763 713 consensus_peers.cc:487] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:34.729260 713 leader_election.cc:343] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.729511 713 consensus_peers.cc:487] T 4a2f72b227e14e46983c906520c29fa3 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:34.730587 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet f4bb74391a1c4e2792226d259a07f977 (table default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]) successfully deleted
I20250625 07:52:34.731663 2286 raft_consensus.cc:491] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.731833 2286 raft_consensus.cc:513] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.732421 2286 leader_election.cc:290] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.732904 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a41ffd566593499299aba5513ccb2f30" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.733175 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a41ffd566593499299aba5513ccb2f30" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:34.733563 850 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:34.733664 851 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.733780 851 leader_election.cc:304] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.734067 2112 raft_consensus.cc:2747] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.737731 1166 ts_tablet_manager.cc:1918] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.738399 1166 log.cc:1199] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/6e5a207d515449e298936d8bb532437a
I20250625 07:52:34.738889 1166 ts_tablet_manager.cc:1939] T 6e5a207d515449e298936d8bb532437a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.740331 1166 tablet_replica.cc:331] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.740615 1160 tablet_replica.cc:331] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.741022 1160 raft_consensus.cc:2241] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.741483 1160 raft_consensus.cc:2270] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.742413 1160 ts_tablet_manager.cc:1905] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.743393 1166 raft_consensus.cc:2241] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.743177 1162 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.744973 1162 ts_tablet_manager.cc:1397] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.003s
I20250625 07:52:34.745146 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6b51e4df123348e6a217afc8bdbf1d66" } new_table_name: "default.Table_83efa7a2252740f7b29059fa6629975a" modify_external_catalogs: true
I20250625 07:52:34.745718 1162 raft_consensus.cc:357] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.745862 1162 raft_consensus.cc:383] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.745909 1162 raft_consensus.cc:738] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.746122 1162 consensus_queue.cc:260] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.751209 1162 ts_tablet_manager.cc:1428] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250625 07:52:34.754237 1162 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.754670 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 110fe299c7c44df69f459da72a1138a5 (table default.Table_55794188bd9d47d29bc77e1c1498512f [id=d281de0430994f13a49c3d53c062dec4]) successfully deleted
I20250625 07:52:34.756099 1162 tablet_bootstrap.cc:654] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.756635 1166 raft_consensus.cc:2270] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.758338 1166 ts_tablet_manager.cc:1905] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.758567 1158 tablet_bootstrap.cc:492] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.758725 1158 ts_tablet_manager.cc:1397] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250625 07:52:34.759500 1158 raft_consensus.cc:357] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.759682 1158 raft_consensus.cc:383] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.759752 1158 raft_consensus.cc:738] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:34.759843 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e0531838d7f4ceba54697f571889b52 has no consensus state
I20250625 07:52:34.759954 1158 consensus_queue.cc:260] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.760471 1158 ts_tablet_manager.cc:1428] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:34.760798 1158 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.763571 1158 tablet_bootstrap.cc:654] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.768883 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet cbdbd9230210455ba615d51f1a98267b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.778458 1160 ts_tablet_manager.cc:1918] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.779592 1161 ts_tablet_manager.cc:1918] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.779733 1161 log.cc:1199] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b1422987deca41a79f8db666d6961643
I20250625 07:52:34.780122 1161 ts_tablet_manager.cc:1939] T b1422987deca41a79f8db666d6961643 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:34.782660 1166 ts_tablet_manager.cc:1918] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.783448 1160 log.cc:1199] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:34.783816 1160 ts_tablet_manager.cc:1939] T 07a07af928ff4d3d894b074ef0cf44eb P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.784940 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 07a07af928ff4d3d894b074ef0cf44eb (table default.Table_eb85d0d01edc400eabcdd8a8175dfbf9 [id=68ab97279148448991b34aeef51de39a]) successfully deleted
W20250625 07:52:34.785665 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet c33b8be28b684f4babfa27e829bb64cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33b8be28b684f4babfa27e829bb64cf already in progress: creating tablet
I20250625 07:52:34.783288 1162 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.786032 1162 ts_tablet_manager.cc:1397] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20250625 07:52:34.786870 1162 raft_consensus.cc:357] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.787089 1162 raft_consensus.cc:383] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.787181 1162 raft_consensus.cc:738] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.787496 1162 consensus_queue.cc:260] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.790143 1162 ts_tablet_manager.cc:1428] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:34.790472 1162 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.792074 1162 tablet_bootstrap.cc:654] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.793222 1166 log.cc:1199] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/97b9a14151fc472e956126620633be32
I20250625 07:52:34.793641 1166 ts_tablet_manager.cc:1939] T 97b9a14151fc472e956126620633be32 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:34.794224 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet af8c114c1c464bcc95dbd287c88976d6 because the tablet was not found. No further retry: Not found: Tablet not found: af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:34.795258 1166 tablet_replica.cc:331] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.795502 1160 tablet_replica.cc:331] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.795751 1160 raft_consensus.cc:2241] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.795946 1160 raft_consensus.cc:2270] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.796489 1160 ts_tablet_manager.cc:1905] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.796743 1166 raft_consensus.cc:2241] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.796927 1166 raft_consensus.cc:2270] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.798527 1166 ts_tablet_manager.cc:1905] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.800089 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 768c9551d33346718af2670a01a626b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.805779 2311 catalog_manager.cc:1261] Loaded cluster ID: f1c1a60bf7df437d8f4c2e6d554d8ac6
I20250625 07:52:34.807484 1158 tablet_bootstrap.cc:492] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.807627 1158 ts_tablet_manager.cc:1397] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.021s
I20250625 07:52:34.807665 1163 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.807783 1163 ts_tablet_manager.cc:1397] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.199s user 0.011s sys 0.001s
I20250625 07:52:34.808380 1158 raft_consensus.cc:357] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.808475 1163 raft_consensus.cc:357] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.808576 1158 raft_consensus.cc:383] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.808655 1163 raft_consensus.cc:383] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.808645 1158 raft_consensus.cc:738] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.808741 1163 raft_consensus.cc:738] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.808928 1158 consensus_queue.cc:260] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.808964 1163 consensus_queue.cc:260] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.809409 1163 ts_tablet_manager.cc:1428] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:34.812983 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d4d367cc123e4bbda8054d1d61d29be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.813350 746 tablet_service.cc:1515] Processing DeleteTablet for tablet b74d8f0eabf1410fadaadb98344b3f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.813688 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:34.814126 713 leader_election.cc:304] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.814451 2288 raft_consensus.cc:2802] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:34.814625 2288 raft_consensus.cc:695] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:34.814899 2288 consensus_queue.cc:237] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.815966 1158 ts_tablet_manager.cc:1428] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:34.816223 1158 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.818766 2178 raft_consensus.cc:491] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.818876 2178 raft_consensus.cc:513] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.819458 2178 leader_election.cc:290] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.821806 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet be7b755d95b14ecea7a691003455f381 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:34.822247 986 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:34.824589 1158 tablet_bootstrap.cc:654] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.828970 2311 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: loading cluster ID for follower catalog manager: success
I20250625 07:52:34.830273 1160 ts_tablet_manager.cc:1918] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.830406 1160 log.cc:1199] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/716e0c4dab584788bbb7631b4ca934bf
I20250625 07:52:34.830709 1160 ts_tablet_manager.cc:1939] T 716e0c4dab584788bbb7631b4ca934bf P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.831528 1166 ts_tablet_manager.cc:1918] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.831640 1166 log.cc:1199] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c0cc3ba09d5e4c3d88e74fffc97eea8b
I20250625 07:52:34.831929 1166 ts_tablet_manager.cc:1939] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.832401 1162 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.832536 1162 ts_tablet_manager.cc:1397] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250625 07:52:34.833287 1162 raft_consensus.cc:357] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.833534 1162 raft_consensus.cc:383] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.833715 1162 raft_consensus.cc:738] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.833973 1162 consensus_queue.cc:260] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:34.836141 985 leader_election.cc:343] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:34.836272 985 leader_election.cc:304] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.836771 2178 raft_consensus.cc:2747] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.833273 1166 tablet_replica.cc:331] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.842104 1166 raft_consensus.cc:2241] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.842710 1166 raft_consensus.cc:2270] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.843909 1166 ts_tablet_manager.cc:1905] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.845769 2178 raft_consensus.cc:491] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.845909 2178 raft_consensus.cc:513] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.846514 2178 leader_election.cc:290] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:34.847438 986 leader_election.cc:343] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 541c983d64f348539979f74199f79e59
W20250625 07:52:34.848459 985 leader_election.cc:343] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 541c983d64f348539979f74199f79e59
I20250625 07:52:34.848590 985 leader_election.cc:304] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.849001 2178 raft_consensus.cc:2747] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.849232 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 716e0c4dab584788bbb7631b4ca934bf (table default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]) successfully deleted
I20250625 07:52:34.849303 1160 tablet_replica.cc:331] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:34.849536 1160 raft_consensus.cc:2241] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.849704 1160 raft_consensus.cc:2270] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.850265 1160 ts_tablet_manager.cc:1905] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.853785 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5d5d75af69a841d5b3961973eb62dbab" } modify_external_catalogs: true
I20250625 07:52:34.853982 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5d5d75af69a841d5b3961973eb62dbab" } modify_external_catalogs: true
I20250625 07:52:34.857937 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b6579bae985346f9bbb23891c04cbe8b" } new_table_name: "default.Table_483af3005b49459e94b99b27f58ceb1d" modify_external_catalogs: true
I20250625 07:52:34.858464 1163 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.860105 1163 tablet_bootstrap.cc:654] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.862076 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_755e5d8d2b6c4c32bb20ec7903fab9c6" } modify_external_catalogs: true
I20250625 07:52:34.862233 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_755e5d8d2b6c4c32bb20ec7903fab9c6" } modify_external_catalogs: true
I20250625 07:52:34.862432 1166 ts_tablet_manager.cc:1918] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.862540 1166 log.cc:1199] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/fdad7dc1b27e4d87a747642a6d47230c
I20250625 07:52:34.862823 1166 ts_tablet_manager.cc:1939] T fdad7dc1b27e4d87a747642a6d47230c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.863549 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8d97e47e913f4458bb17f7c95671f879" } new_table_name: "default.Table_bba9d21266f54573897e4dd408928ab4" modify_external_catalogs: true
I20250625 07:52:34.864109 1166 tablet_replica.cc:331] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:34.864290 1166 raft_consensus.cc:2241] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.864439 1166 raft_consensus.cc:2270] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.865271 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ae0c597df57144d3b0b8fad6aacfde2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.865728 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0c597df57144d3b0b8fad6aacfde2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.868641 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_747b0d5eb8ca4f1ab30aced4c1431e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.869117 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_747b0d5eb8ca4f1ab30aced4c1431e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.870597 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f326a912662f42e2abb74225853c112d
I20250625 07:52:34.872046 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f326a912662f42e2abb74225853c112d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.872256 1166 ts_tablet_manager.cc:1905] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:34.872562 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet f326a912662f42e2abb74225853c112d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.874852 1161 tablet_replica.cc:331] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:34.874946 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b1422987deca41a79f8db666d6961643 (table default.Table_1b99fa339b63429ca65b10e0fadd2fa3 [id=c91ad15eea194ae98235c1bc169430e4]) successfully deleted
I20250625 07:52:34.875147 1161 raft_consensus.cc:2241] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:34.875326 1161 raft_consensus.cc:2270] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:34.876955 1161 ts_tablet_manager.cc:1905] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:34.877756 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a72e472025348da8510f66904424a2b has no consensus state
I20250625 07:52:34.878919 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dec54e44daf642c295a24542167b57c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.879406 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec54e44daf642c295a24542167b57c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.881100 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f406d1c8777f474c97f6e804cb230bb6
W20250625 07:52:34.882043 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69f0d45815ee45f3b48e4675f7ed901f consensus state has no leader
I20250625 07:52:34.882370 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f406d1c8777f474c97f6e804cb230bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.883826 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet f406d1c8777f474c97f6e804cb230bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.888481 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c96df891fb1343df8cc90ee2304287fb" } modify_external_catalogs: true
I20250625 07:52:34.888646 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c96df891fb1343df8cc90ee2304287fb" } modify_external_catalogs: true
I20250625 07:52:34.890992 1163 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.892215 1163 ts_tablet_manager.cc:1397] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.003s
I20250625 07:52:34.892226 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f426b18fff7f499db91894562dc4736e" } modify_external_catalogs: true
I20250625 07:52:34.892753 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f426b18fff7f499db91894562dc4736e" } modify_external_catalogs: true
I20250625 07:52:34.893337 1163 raft_consensus.cc:357] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.893501 1163 raft_consensus.cc:383] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.893553 1163 raft_consensus.cc:738] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.893033 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_43525db1d2034f6f8460bbfccb691fc0" } new_table_name: "default.Table_fa3e5727a6be492f9e2f0f6a39d36036" modify_external_catalogs: true
I20250625 07:52:34.895035 1166 ts_tablet_manager.cc:1918] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.895077 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_22c365283ad046a7bf006c4f05f78492"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.895373 1163 consensus_queue.cc:260] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:34.895720 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22c365283ad046a7bf006c4f05f78492 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.896063 1163 ts_tablet_manager.cc:1428] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:34.896348 1163 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.895246 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eaf3a4cc1ae44e4494084ce8431f9eaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.897122 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e8c4c4252edf421ba29135cd531dc8ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.897351 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaf3a4cc1ae44e4494084ce8431f9eaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:34.897464 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8c4c4252edf421ba29135cd531dc8ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.897989 1163 tablet_bootstrap.cc:654] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.898357 1166 log.cc:1199] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0aeb7077591f466eb77e3096f6b1d74b
I20250625 07:52:34.898902 1166 ts_tablet_manager.cc:1939] T 0aeb7077591f466eb77e3096f6b1d74b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:34.899341 1162 ts_tablet_manager.cc:1428] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.067s user 0.005s sys 0.017s
I20250625 07:52:34.900843 1162 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.902508 1162 tablet_bootstrap.cc:654] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.902989 1158 tablet_bootstrap.cc:492] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.899654 2346 tablet.cc:1722] T 1405e55703324bc7a1e26fe2c349bda7 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:34.903160 1158 ts_tablet_manager.cc:1397] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.010s
W20250625 07:52:34.904052 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69e2214bf6e54052bf8357b32caf17e6 consensus state has no leader
I20250625 07:52:34.906054 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7fd497aa0b8c4838906524f4423c3756
I20250625 07:52:34.908351 2348 raft_consensus.cc:491] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.908716 2348 raft_consensus.cc:513] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.909202 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fd497aa0b8c4838906524f4423c3756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:34.909721 2348 leader_election.cc:290] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.910147 2286 raft_consensus.cc:491] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.910267 2286 raft_consensus.cc:513] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.910810 2286 leader_election.cc:290] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:34.911617 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.912011 1163 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.912125 1163 ts_tablet_manager.cc:1397] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
W20250625 07:52:34.912909 850 leader_election.cc:343] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.910130 2346 tablet.cc:1722] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:34.913558 2178 raft_consensus.cc:491] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.913703 2178 raft_consensus.cc:513] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.914297 2178 leader_election.cc:290] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:34.914836 1158 raft_consensus.cc:357] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.915000 1158 raft_consensus.cc:383] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.915058 1158 raft_consensus.cc:738] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.915242 1158 consensus_queue.cc:260] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.915686 1158 ts_tablet_manager.cc:1428] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20250625 07:52:34.915992 1158 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.915870 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9c021cd55d3e4dd5aacc7954f40aff2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.916416 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c021cd55d3e4dd5aacc7954f40aff2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.916561 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fd497aa0b8c4838906524f4423c3756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.916906 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aa08f96a75714b51bcde3e7b3dcf2c77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.917351 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa08f96a75714b51bcde3e7b3dcf2c77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.920059 1163 raft_consensus.cc:357] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.920138 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5c86dbb25a2d445aa635a0dcf9043b9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:34.920717 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c86dbb25a2d445aa635a0dcf9043b9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.921079 2178 raft_consensus.cc:491] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.921247 1163 raft_consensus.cc:383] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.921239 2178 raft_consensus.cc:513] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.921320 1163 raft_consensus.cc:738] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.921535 1163 consensus_queue.cc:260] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:34.921809 2178 leader_election.cc:290] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.922029 1163 ts_tablet_manager.cc:1428] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20250625 07:52:34.922616 985 leader_election.cc:343] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2771159858fc41a1ad930229986edd92
I20250625 07:52:34.924136 1162 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.924316 1162 ts_tablet_manager.cc:1397] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20250625 07:52:34.925082 1162 raft_consensus.cc:357] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.925292 1162 raft_consensus.cc:383] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.925395 1162 raft_consensus.cc:738] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.925617 1162 consensus_queue.cc:260] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:34.925772 985 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:34.926107 1162 ts_tablet_manager.cc:1428] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.926369 1162 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.926293 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4dc958f1088e4e719b8afc3b90c3b8dc" } new_table_name: "default.Table_87581c07607f4640ba541cd020c87c34" modify_external_catalogs: true
I20250625 07:52:34.927953 1162 tablet_bootstrap.cc:654] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.929401 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:34.929697 1158 tablet_bootstrap.cc:654] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.929736 1163 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:34.929865 986 leader_election.cc:343] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2771159858fc41a1ad930229986edd92
I20250625 07:52:34.930009 986 leader_election.cc:304] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.930498 2178 raft_consensus.cc:2747] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.931288 1163 tablet_bootstrap.cc:654] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:34.931973 986 leader_election.cc:343] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:34.932092 986 leader_election.cc:304] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.932655 2178 raft_consensus.cc:2747] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:34.933152 851 leader_election.cc:343] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:34.933274 851 leader_election.cc:304] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:34.933878 2286 raft_consensus.cc:2747] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.934616 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e60d4f9aed44082b9905fbb44ffb110 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:34.936779 1162 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.936964 1162 ts_tablet_manager.cc:1397] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250625 07:52:34.937801 1162 raft_consensus.cc:357] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.938030 1162 raft_consensus.cc:383] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.938131 1162 raft_consensus.cc:738] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.938361 1162 consensus_queue.cc:260] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.938553 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_672d41abf5214dcdbd0c430131712062"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:34.939131 1162 ts_tablet_manager.cc:1428] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 07:52:34.939179 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_672d41abf5214dcdbd0c430131712062 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:34.939499 1162 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.941094 1162 tablet_bootstrap.cc:654] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.943524 1163 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.943724 1163 ts_tablet_manager.cc:1397] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250625 07:52:34.944552 1163 raft_consensus.cc:357] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.944810 1163 raft_consensus.cc:383] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.944909 1163 raft_consensus.cc:738] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.945181 1163 consensus_queue.cc:260] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.945765 1163 ts_tablet_manager.cc:1428] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.946067 1163 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.947690 1163 tablet_bootstrap.cc:654] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.949478 1162 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.949663 1162 ts_tablet_manager.cc:1397] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250625 07:52:34.950048 2178 raft_consensus.cc:491] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:34.950208 2178 raft_consensus.cc:513] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.950403 1162 raft_consensus.cc:357] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.950552 1162 raft_consensus.cc:383] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.950601 1162 raft_consensus.cc:738] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.950768 1162 consensus_queue.cc:260] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.951220 1162 ts_tablet_manager.cc:1428] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250625 07:52:34.951316 2178 leader_election.cc:290] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:34.951676 1162 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:34.952255 985 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1898ff5b7626478bb997f7a433477498
W20250625 07:52:34.953068 986 leader_election.cc:343] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 1898ff5b7626478bb997f7a433477498
I20250625 07:52:34.953187 986 leader_election.cc:304] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:34.953704 2178 raft_consensus.cc:2747] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:34.953825 1162 tablet_bootstrap.cc:654] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.957868 1163 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.958055 1163 ts_tablet_manager.cc:1397] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20250625 07:52:34.959211 1163 raft_consensus.cc:357] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.959444 1163 raft_consensus.cc:383] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.959548 1163 raft_consensus.cc:738] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.959820 1163 consensus_queue.cc:260] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.960399 1163 ts_tablet_manager.cc:1428] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:34.960729 1163 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.961809 1158 tablet_bootstrap.cc:492] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:34.962036 1158 ts_tablet_manager.cc:1397] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.004s
I20250625 07:52:34.962428 1163 tablet_bootstrap.cc:654] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.962924 1158 raft_consensus.cc:357] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.963187 1158 raft_consensus.cc:383] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.963311 1158 raft_consensus.cc:738] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.963619 1158 consensus_queue.cc:260] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.964280 1158 ts_tablet_manager.cc:1428] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:34.964745 1162 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.964931 1162 ts_tablet_manager.cc:1397] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20250625 07:52:34.965350 1158 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:34.968757 1162 raft_consensus.cc:357] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.969002 1162 raft_consensus.cc:383] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.969106 1162 raft_consensus.cc:738] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.969360 1162 consensus_queue.cc:260] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.970009 1162 ts_tablet_manager.cc:1428] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250625 07:52:34.970563 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_586f18dcb3f5429bbb79072497975a59" } new_table_name: "default.Table_e2eca35a90b04de6bc95ef311484b664" modify_external_catalogs: true
I20250625 07:52:34.971418 1158 tablet_bootstrap.cc:654] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:34.972223 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d18ad92511024e1daab6cca07e5e06b5 consensus state has no leader
I20250625 07:52:34.972816 1162 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:34.974584 1162 tablet_bootstrap.cc:654] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.978085 1163 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.978356 1163 ts_tablet_manager.cc:1397] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250625 07:52:34.979166 1163 raft_consensus.cc:357] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.979395 1163 raft_consensus.cc:383] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.979501 1163 raft_consensus.cc:738] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.979774 1163 consensus_queue.cc:260] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.980294 1163 ts_tablet_manager.cc:1428] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.980604 1163 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:34.982196 1163 tablet_bootstrap.cc:654] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.982957 1160 ts_tablet_manager.cc:1918] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:34.983070 1160 log.cc:1199] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7db12504a1a54add920561dae27d2d2b
I20250625 07:52:34.983433 1160 ts_tablet_manager.cc:1939] T 7db12504a1a54add920561dae27d2d2b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:34.984354 1162 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:34.984326 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7db12504a1a54add920561dae27d2d2b (table default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]) successfully deleted
I20250625 07:52:34.984583 1162 ts_tablet_manager.cc:1397] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20250625 07:52:34.985430 1162 raft_consensus.cc:357] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.985657 1162 raft_consensus.cc:383] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:34.985757 1162 raft_consensus.cc:738] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:34.985982 1162 consensus_queue.cc:260] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:34.986593 1162 ts_tablet_manager.cc:1428] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:34.987195 1162 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:34.988116 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 833cb1c75c5e429dac3428784007ab40 consensus state has no leader
I20250625 07:52:34.988807 590 master_service.cc:948] ReplaceTablet: received request to replace tablet d9b1ec100ca4488c9d7fc470246ffe47 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:34.988920 1162 tablet_bootstrap.cc:654] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:34.990890 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet f191436f3b9244eaa12d211b6f125175 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e77ff1b8ca848ff8e1b4538645b4be2) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:34.990967 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 170866us
I20250625 07:52:34.991536 824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38477
I20250625 07:52:34.993346 2340 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:34.995487 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 6e5a207d515449e298936d8bb532437a (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
W20250625 07:52:34.996248 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bff347d838fe4e56a195a957d0deb742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff347d838fe4e56a195a957d0deb742 already in progress: creating tablet
W20250625 07:52:34.996912 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bc387934e5204a3e9ce6f13d1dc3f777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387934e5204a3e9ce6f13d1dc3f777 already in progress: creating tablet
I20250625 07:52:34.996997 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c263b671be384cd097041c60306034de" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:34.997433 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 97b9a14151fc472e956126620633be32 (table default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]) successfully deleted
I20250625 07:52:34.997942 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (table default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]) successfully deleted
I20250625 07:52:34.998405 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet fdad7dc1b27e4d87a747642a6d47230c (table default.Table_ba10e2f968a543d4a5173431533d7e72 [id=6da9e5a92547450b83830b27727c69ef]) successfully deleted
I20250625 07:52:34.998612 912 raft_consensus.cc:1273] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:34.998770 1163 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:34.998890 1163 ts_tablet_manager.cc:1397] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
W20250625 07:52:34.998891 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 768c9551d33346718af2670a01a626b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 768c9551d33346718af2670a01a626b0 already in progress: creating tablet
W20250625 07:52:34.999094 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d4d367cc123e4bbda8054d1d61d29be1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4d367cc123e4bbda8054d1d61d29be1 already in progress: creating tablet
W20250625 07:52:34.999735 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b74d8f0eabf1410fadaadb98344b3f62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74d8f0eabf1410fadaadb98344b3f62 already in progress: creating tablet
I20250625 07:52:35.000118 1163 raft_consensus.cc:357] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.000196 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0aeb7077591f466eb77e3096f6b1d74b (table default.Table_1225d003be2b4d048266736844a78a01 [id=357521ce57d54dafae9cfd2efbe731c2]) successfully deleted
W20250625 07:52:35.001523 713 consensus_peers.cc:487] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:35.001777 713 leader_election.cc:343] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.001327 1163 raft_consensus.cc:383] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.002560 1163 raft_consensus.cc:738] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.002903 1163 consensus_queue.cc:260] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.003743 1163 ts_tablet_manager.cc:1428] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:35.004539 1163 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.006444 746 tablet_service.cc:1515] Processing DeleteTablet for tablet f326a912662f42e2abb74225853c112d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.007140 746 tablet_service.cc:1515] Processing DeleteTablet for tablet f406d1c8777f474c97f6e804cb230bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.007403 1163 tablet_bootstrap.cc:654] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.017995 1163 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.019435 1163 ts_tablet_manager.cc:1397] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20250625 07:52:35.020881 1163 raft_consensus.cc:357] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.021111 1163 raft_consensus.cc:383] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.021209 1163 raft_consensus.cc:738] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.021468 1163 consensus_queue.cc:260] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.022035 1163 ts_tablet_manager.cc:1428] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.023492 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fd497aa0b8c4838906524f4423c3756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.023800 1166 tablet_replica.cc:331] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:35.024052 1166 raft_consensus.cc:2241] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:35.024389 1166 raft_consensus.cc:2270] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.025110 1166 ts_tablet_manager.cc:1905] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.030978 1158 tablet_bootstrap.cc:492] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.031230 1158 ts_tablet_manager.cc:1397] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.005s
I20250625 07:52:35.032262 1158 raft_consensus.cc:357] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.032646 1158 raft_consensus.cc:383] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.033214 1158 raft_consensus.cc:738] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.033623 1158 consensus_queue.cc:260] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.034313 1158 ts_tablet_manager.cc:1428] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:35.034371 1166 ts_tablet_manager.cc:1918] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:35.034478 1166 log.cc:1199] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7fd497aa0b8c4838906524f4423c3756
I20250625 07:52:35.034814 1166 ts_tablet_manager.cc:1939] T 7fd497aa0b8c4838906524f4423c3756 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:35.036023 1162 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.036203 1162 ts_tablet_manager.cc:1397] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.002s
I20250625 07:52:35.037024 1162 raft_consensus.cc:357] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.037254 1162 raft_consensus.cc:383] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.037350 1162 raft_consensus.cc:738] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.037623 1162 consensus_queue.cc:260] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:35.037985 713 leader_election.cc:343] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c263b671be384cd097041c60306034de
I20250625 07:52:35.038100 713 leader_election.cc:304] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.038362 2342 consensus_queue.cc:1035] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:35.038430 2339 consensus_queue.cc:1035] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:35.038655 2288 raft_consensus.cc:2747] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.039135 2346 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:35.039537 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b9491eb269948a29888a7301ea75d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:35.041289 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f326a912662f42e2abb74225853c112d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326a912662f42e2abb74225853c112d already in progress: creating tablet
W20250625 07:52:35.041517 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f406d1c8777f474c97f6e804cb230bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f406d1c8777f474c97f6e804cb230bb6 already in progress: creating tablet
I20250625 07:52:35.041738 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]) successfully deleted
I20250625 07:52:35.044903 1158 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.050500 1161 ts_tablet_manager.cc:1918] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.050772 1161 log.cc:1199] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c8c015c202544914b74f1e245b496f82
I20250625 07:52:35.051151 1161 ts_tablet_manager.cc:1939] T c8c015c202544914b74f1e245b496f82 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.051975 1162 ts_tablet_manager.cc:1428] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250625 07:52:35.052080 1161 tablet_replica.cc:331] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.052294 1161 raft_consensus.cc:2241] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.052549 1161 raft_consensus.cc:2270] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.052914 1162 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.054155 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c8c015c202544914b74f1e245b496f82 (table default.Table_2b519d271b96497dafe0b274a8755066 [id=4a285d0804ea403bb60aa49bfde4bd9a]) successfully deleted
I20250625 07:52:35.058097 1162 tablet_bootstrap.cc:654] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.061002 2370 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:35.061817 1158 tablet_bootstrap.cc:654] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.069681 1163 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.073711 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8319e385441549e98fc4f6f600dd17b7" } modify_external_catalogs: true
I20250625 07:52:35.073976 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8319e385441549e98fc4f6f600dd17b7" } modify_external_catalogs: true
I20250625 07:52:35.074276 1162 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.077250 1162 ts_tablet_manager.cc:1397] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20250625 07:52:35.078442 1163 tablet_bootstrap.cc:654] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.079299 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c6761e13384642f19befe5477a330593" } new_table_name: "default.Table_2d3b375ead844126bcf8371bf41c071a" modify_external_catalogs: true
I20250625 07:52:35.080210 1162 raft_consensus.cc:357] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.080356 1162 raft_consensus.cc:383] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.080412 1162 raft_consensus.cc:738] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.082652 1162 consensus_queue.cc:260] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.082890 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_46671c5a6dee4493836a690b730763e9" } modify_external_catalogs: true
I20250625 07:52:35.083065 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_46671c5a6dee4493836a690b730763e9" } modify_external_catalogs: true
I20250625 07:52:35.083546 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_86b56ad960e1457ca74b3db973f6cbd0" } new_table_name: "default.Table_49733a5aa75644579663a60ccfd9118f" modify_external_catalogs: true
I20250625 07:52:35.085808 1162 ts_tablet_manager.cc:1428] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:35.085850 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b3a3855c16d64eeabd95e1bd8de74d2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.086421 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3a3855c16d64eeabd95e1bd8de74d2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.086901 1162 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.088528 1162 tablet_bootstrap.cc:654] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:35.092029 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7c6d068d2324ea6a07bb9be74578870 consensus state has no leader
I20250625 07:52:35.090442 2348 raft_consensus.cc:491] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.092841 2348 raft_consensus.cc:513] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.093429 2348 leader_election.cc:290] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.094127 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.095135 713 leader_election.cc:343] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.095250 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.095664 912 raft_consensus.cc:2466] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.096057 713 leader_election.cc:304] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.096359 2348 raft_consensus.cc:2802] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.096453 2348 raft_consensus.cc:491] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.096516 2348 raft_consensus.cc:3058] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.099225 2348 raft_consensus.cc:513] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.099751 2348 leader_election.cc:290] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.100590 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:35.100792 2112 raft_consensus.cc:491] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.100925 2112 raft_consensus.cc:513] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.101480 2112 leader_election.cc:290] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.101961 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da895c90d6ee40379fe8181c2dbd7a1e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.102123 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da895c90d6ee40379fe8181c2dbd7a1e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.102432 850 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.102571 1161 ts_tablet_manager.cc:1905] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:35.102850 851 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.102973 851 leader_election.cc:304] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.103179 2311 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: acquiring CA information for follower catalog manager: success
W20250625 07:52:35.103292 713 leader_election.cc:343] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.103624 2112 raft_consensus.cc:2747] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.103907 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.104065 912 raft_consensus.cc:3058] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.107656 912 raft_consensus.cc:2466] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.108397 713 leader_election.cc:304] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.108664 2348 raft_consensus.cc:2802] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.108801 2348 raft_consensus.cc:695] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.109014 2348 consensus_queue.cc:237] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.113694 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_99d809187ffc4bc2922585193e957568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.114540 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99d809187ffc4bc2922585193e957568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:35.121237 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36c61f71a1674957805a48ccd6d0d4b3 consensus state has no leader
I20250625 07:52:35.121615 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_06ea495eb1d048f0a6cba1913dcc5406"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.122097 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ea495eb1d048f0a6cba1913dcc5406 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.123513 2112 raft_consensus.cc:491] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.123672 2112 raft_consensus.cc:513] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.124256 2112 leader_election.cc:290] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.124830 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f1d251171314118b66e3a921a4a50c5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.124961 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f1d251171314118b66e3a921a4a50c5" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.125393 850 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.125548 851 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.125706 851 leader_election.cc:304] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.126214 2112 raft_consensus.cc:2747] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.127784 1162 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.128551 1162 ts_tablet_manager.cc:1397] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.003s
I20250625 07:52:35.129698 1162 raft_consensus.cc:357] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.131717 1162 raft_consensus.cc:383] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.132018 1162 raft_consensus.cc:738] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.132440 1162 consensus_queue.cc:260] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.133525 2178 raft_consensus.cc:491] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.133736 2178 raft_consensus.cc:513] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.134049 1162 ts_tablet_manager.cc:1428] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:35.134445 2178 leader_election.cc:290] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.137491 986 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3b9491eb269948a29888a7301ea75d42
I20250625 07:52:35.140990 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_895827fb99994402b75be290df98a2ff" } new_table_name: "default.Table_c4fe0c9ea95b470b93d84500fbd74b69" modify_external_catalogs: true
W20250625 07:52:35.145236 985 leader_election.cc:343] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3b9491eb269948a29888a7301ea75d42
I20250625 07:52:35.145411 985 leader_election.cc:304] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.146152 2178 raft_consensus.cc:2747] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.149456 1162 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.151839 1162 tablet_bootstrap.cc:654] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.152393 2370 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:35.153599 1161 ts_tablet_manager.cc:1918] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.153739 1161 log.cc:1199] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/fcc3afc210a540058294121d2b83c66d
I20250625 07:52:35.154138 1161 ts_tablet_manager.cc:1939] T fcc3afc210a540058294121d2b83c66d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.155002 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1175) took 1009 ms. Trace:
I20250625 07:52:35.155158 1161 rpcz_store.cc:276] 0625 07:52:34.145459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.145526 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:35.051999 (+906473us) ts_tablet_manager.cc:1145] Deleting tablet fcc3afc210a540058294121d2b83c66d
0625 07:52:35.052008 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.107622 (+ 55614us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.153445 (+ 45823us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.154960 (+ 1515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":906321}
I20250625 07:52:35.155630 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet fcc3afc210a540058294121d2b83c66d (table default.Table_bb831997b8cf4b99a7779a4a6b23a238 [id=8a01186fa596413fb75529d0d57a8d9d]) successfully deleted
I20250625 07:52:35.156793 1158 tablet_bootstrap.cc:492] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.156965 1158 ts_tablet_manager.cc:1397] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.004s
I20250625 07:52:35.157537 2112 raft_consensus.cc:491] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.157703 2112 raft_consensus.cc:513] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.157877 1158 raft_consensus.cc:357] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.158079 1158 raft_consensus.cc:383] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.158169 1158 raft_consensus.cc:738] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.158430 1158 consensus_queue.cc:260] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.158533 2112 leader_election.cc:290] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.158852 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.159132 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.159402 850 leader_election.cc:343] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.159647 851 leader_election.cc:343] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.159765 851 leader_election.cc:304] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.160210 2112 raft_consensus.cc:2747] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.163609 2178 raft_consensus.cc:491] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.163779 2178 raft_consensus.cc:513] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.164315 2178 leader_election.cc:290] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:35.165575 986 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:35.169034 1163 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:35.169100 985 leader_election.cc:343] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:35.169256 1163 ts_tablet_manager.cc:1397] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.004s
I20250625 07:52:35.169242 985 leader_election.cc:304] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.169708 2178 raft_consensus.cc:2747] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.170219 1163 raft_consensus.cc:357] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.171186 1163 raft_consensus.cc:383] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.171320 1163 raft_consensus.cc:738] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.171605 1163 consensus_queue.cc:260] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.172130 1163 ts_tablet_manager.cc:1428] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:35.172796 1158 ts_tablet_manager.cc:1428] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250625 07:52:35.173225 1163 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.183033 1158 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.185235 1158 tablet_bootstrap.cc:654] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.185384 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d92c6481d56745f9a67a5c0cdf374c9d" } modify_external_catalogs: true
I20250625 07:52:35.185607 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d92c6481d56745f9a67a5c0cdf374c9d" } modify_external_catalogs: true
W20250625 07:52:35.191277 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b2aa7a64bfd49bc8e3accf840c66a4c consensus state has no leader
I20250625 07:52:35.191493 1162 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.191625 1162 ts_tablet_manager.cc:1397] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.010s
I20250625 07:52:35.192370 1162 raft_consensus.cc:357] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.192552 1162 raft_consensus.cc:383] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.192607 1162 raft_consensus.cc:738] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.192850 1162 consensus_queue.cc:260] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.193471 1162 ts_tablet_manager.cc:1428] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.193743 1162 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.195355 1162 tablet_bootstrap.cc:654] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.197284 2348 raft_consensus.cc:491] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.197419 2348 raft_consensus.cc:513] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.197955 2348 leader_election.cc:290] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.198932 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.198947 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.199144 912 raft_consensus.cc:2466] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:35.199648 713 leader_election.cc:343] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.199811 713 leader_election.cc:304] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.200210 2348 raft_consensus.cc:2802] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.200296 2348 raft_consensus.cc:491] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.200350 2348 raft_consensus.cc:3058] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:34.505970 1908 rpcz_store.cc:276] 0625 07:52:29.484907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:29.484965 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:32.473066 (+2988101us) raft_consensus.cc:1469] Updating replica for 478 ops
0625 07:52:32.706778 (+233712us) raft_consensus.cc:1532] Early marking committed up to index 721
0625 07:52:32.757191 (+ 50413us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:32.757200 (+ 9us) write_op.cc:454] Released schema lock
0625 07:52:32.760958 (+ 3758us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:32.760964 (+ 6us) write_op.cc:454] Released schema lock
0625 07:52:32.772406 (+ 11442us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:32.772413 (+ 7us) write_op.cc:454] Released schema lock
0625 07:52:32.837361 (+ 64948us) raft_consensus.cc:1537] Triggering prepare for 478 ops
0625 07:52:32.837796 (+ 435us) write_op.cc:270] Start()
0625 07:52:32.837847 (+ 51us) write_op.cc:276] Timestamp: P: 1750837941597555 usec, L: 0
0625 07:52:32.847576 (+ 9729us) write_op.cc:270] Start()
0625 07:52:32.847616 (+ 40us) write_op.cc:276] Timestamp: P: 1750837941598453 usec, L: 0
0625 07:52:32.847814 (+ 198us) write_op.cc:270] Start()
0625 07:52:32.847834 (+ 20us) write_op.cc:276] Timestamp: P: 1750837941599313 usec, L: 0
0625 07:52:32.847970 (+ 136us) write_op.cc:270] Start()
0625 07:52:32.847988 (+ 18us) write_op.cc:276] Timestamp: P: 1750837941709056 usec, L: 0
0625 07:52:32.848114 (+ 126us) write_op.cc:270] Start()
0625 07:52:32.848130 (+ 16us) write_op.cc:276] Timestamp: P: 1750837941709967 usec, L: 0
0625 07:52:32.848237 (+ 107us) write_op.cc:270] Start()
0625 07:52:32.848252 (+ 15us) write_op.cc:276] Timestamp: P: 1750837941710662 usec, L: 0
0625 07:52:32.848410 (+ 158us) write_op.cc:270] Start()
0625 07:52:32.848436 (+ 26us) write_op.cc:276] Timestamp: P: 1750837941711156 usec, L: 0
0625 07:52:32.848590 (+ 154us) write_op.cc:270] Start()
0625 07:52:32.848611 (+ 21us) write_op.cc:276] Timestamp: P: 1750837941711699 usec, L: 0
0625 07:52:32.855394 (+ 6783us) write_op.cc:270] Start()
0625 07:52:32.855597 (+ 203us) write_op.cc:276] Timestamp: P: 1750837941712163 usec, L: 0
0625 07:52:32.855834 (+ 237us) write_op.cc:270] Start()
0625 07:52:32.855863 (+ 29us) write_op.cc:276] Timestamp: P: 1750837941712760 usec, L: 0
0625 07:52:32.856011 (+ 148us) write_op.cc:270] Start()
0625 07:52:32.856034 (+ 23us) write_op.cc:276] Timestamp: P: 1750837941713294 usec, L: 0
0625 07:52:32.856176 (+ 142us) write_op.cc:270] Start()
0625 07:52:32.856198 (+ 22us) write_op.cc:276] Timestamp: P: 1750837941713814 usec, L: 0
0625 07:52:32.856361 (+ 163us) write_op.cc:270] Start()
0625 07:52:32.856384 (+ 23us) write_op.cc:276] Timestamp: P: 1750837941714386 usec, L: 0
0625 07:52:32.856534 (+ 150us) write_op.cc:270] Start()
0625 07:52:32.856559 (+ 25us) write_op.cc:276] Timestamp: P: 1750837941760483 usec, L: 0
0625 07:52:32.856725 (+ 166us) write_op.cc:270] Start()
0625 07:52:32.856749 (+ 24us) write_op.cc:276] Timestamp: P: 1750837941762171 usec, L: 0
0625 07:52:32.857055 (+ 306us) write_op.cc:270] Start()
0625 07:52:32.857079 (+ 24us) write_op.cc:276] Timestamp: P: 1750837941763981 usec, L: 0
0625 07:52:32.857243 (+ 164us) write_op.cc:270] Start()
0625 07:52:32.857267 (+ 24us) write_op.cc:276] Timestamp: P: 1750837941766210 usec, L: 0
0625 07:52:32.857414 (+ 147us) write_op.cc:270] Start()
0625 07:52:32.857434 (+ 20us) write_op.cc:276] Timestamp: P: 1750837941767432 usec, L: 0
0625 07:52:32.857538 (+ 104us) write_op.cc:270] Start()
0625 07:52:32.857554 (+ 16us) write_op.cc:276] Timestamp: P: 1750837941780860 usec, L: 0
0625 07:52:32.857655 (+ 101us) write_op.cc:270] Start()
0625 07:52:32.857671 (+ 16us) write_op.cc:276] Timestamp: P: 1750837941783044 usec, L: 0
0625 07:52:32.857779 (+ 108us) write_op.cc:270] Start()
0625 07:52:32.857819 (+ 40us) write_op.cc:276] Timestamp: P: 1750837941789944 usec, L: 0
0625 07:52:32.857934 (+ 115us) write_op.cc:270] Start()
0625 07:52:32.857950 (+ 16us) write_op.cc:276] Timestamp: P: 1750837941790871 usec, L: 0
0625 07:52:32.858100 (+ 150us) write_op.cc:270] Start()
0625 07:52:32.858119 (+ 19us) write_op.cc:276] Timestamp: P: 1750837941804025 usec, L: 0
0625 07:52:32.858269 (+ 150us) write_op.cc:270] Start()
0625 07:52:32.858295 (+ 26us) write_op.cc:276] Timestamp: P: 1750837941804609 usec, L: 0
0625 07:52:32.858444 (+ 149us) write_op.cc:270] Start()
0625 07:52:32.858468 (+ 24us) write_op.cc:276] Timestamp: P: 1750837941821092 usec, L: 0
0625 07:52:32.858613 (+ 145us) write_op.cc:270] Start()
0625 07:52:32.858638 (+ 25us) write_op.cc:276] Timestamp: P: 1750837941828480 usec, L: 0
0625 07:52:32.858804 (+ 166us) write_op.cc:270] Start()
0625 07:52:32.858827 (+ 23us) write_op.cc:276] Timestamp: P: 1750837941940193 usec, L: 0
0625 07:52:32.858980 (+ 153us) write_op.cc:270] Start()
0625 07:52:32.859003 (+ 23us) write_op.cc:276] Timestamp: P: 1750837941941790 usec, L: 0
0625 07:52:32.859147 (+ 144us) write_op.cc:270] Start()
0625 07:52:32.859169 (+ 22us) write_op.cc:276] Timestamp: P: 1750837941959099 usec, L: 0
0625 07:52:32.859286 (+ 117us) write_op.cc:270] Start()
0625 07:52:32.859303 (+ 17us) write_op.cc:276] Timestamp: P: 1750837941960009 usec, L: 0
0625 07:52:32.859443 (+ 140us) write_op.cc:270] Start()
0625 07:52:32.859468 (+ 25us) write_op.cc:276] Timestamp: P: 1750837941974377 usec, L: 0
0625 07:52:32.859622 (+ 154us) write_op.cc:270] Start()
0625 07:52:32.859645 (+ 23us) write_op.cc:276] Timestamp: P: 1750837941995421 usec, L: 0
0625 07:52:32.859841 (+ 196us) write_op.cc:270] Start()
0625 07:52:32.859865 (+ 24us) write_op.cc:276] Timestamp: P: 1750837941999234 usec, L: 0
0625 07:52:32.860035 (+ 170us) write_op.cc:270] Start()
0625 07:52:32.860058 (+ 23us) write_op.cc:276] Timestamp: P: 1750837942005885 usec, L: 0
0625 07:52:32.860198 (+ 140us) write_op.cc:270] Start()
0625 07:52:32.860221 (+ 23us) write_op.cc:276] Timestamp: P: 1750837942008758 usec, L: 0
0625 07:52:32.860351 (+ 130us) write_op.cc:270] Start()
0625 07:52:32.860367 (+ 16us) write_op.cc:276] Timestamp: P: 1750837942009357 usec, L: 0
0625 07:52:32.860480 (+ 113us) write_op.cc:270] Start()
0625 07:52:32.860496 (+ 16us) write_op.cc:276] Timestamp: P: 1750837942012418 usec, L: 0
0625 07:52:32.860604 (+ 108us) write_op.cc:270] Start()
0625 07:52:32.860621 (+ 17us) write_op.cc:276] Timestamp: P: 1750837942014417 usec, L: 0
0625 07:52:32.957673 (+ 97052us) write_op.cc:270] Start()
0625 07:52:32.957705 (+ 32us) write_op.cc:276] Timestamp: P: 1750837942125949 usec, L: 0
0625 07:52:32.957915 (+ 210us) write_op.cc:270] Start()
0625 07:52:32.957938 (+ 23us) write_op.cc:276] Timestamp: P: 1750837942132799 usec, L: 0
0625 07:52:32.958092 (+ 154us) write_op.cc:270] Start()
0625 07:52:32.958113 (+ 21us) write_op.cc:276] Timestamp: P: 1750837942133418 usec, L: 0
0625 07:52:32.958260 (+ 147us) write_op.cc:270] Start()
0625 07:52:32.958285 (+ 25us) write_op.cc:276] Timestamp: P: 1750837942147266 usec, L: 0
0625 07:52:32.958437 (+ 152us) write_op.cc:270] Start()
0625 07:52:32.958458 (+ 21us) write_op.cc:276] Timestamp: P: 1750837942148343 usec, L: 0
0625 07:52:32.958614 (+ 156us) write_op.cc:270] Start()
0625 07:52:32.958635 (+ 21us) write_op.cc:276] Timestamp: P: 1750837942150645 usec, L: 0
0625 07:52:32.958780 (+ 145us) write_op.cc:270] Start()
0625 07:52:32.958800 (+ 20us) write_op.cc:276] Timestamp: P: 1750837942160891 usec, L: 0
0625 07:52:32.958918 (+ 118us) write_op.cc:270] Start()
0625 07:52:32.958935 (+ 17us) write_op.cc:276] Timestamp: P: 1750837942166031 usec, L: 0
0625 07:52:32.959057 (+ 122us) write_op.cc:270] Start()
0625 07:52:32.959075 (+ 18us) write_op.cc:276] Timestamp: P: 1750837942179464 usec, L: 0
0625 07:52:32.959215 (+ 140us) write_op.cc:270] Start()
0625 07:52:32.959233 (+ 18us) write_op.cc:276] Timestamp: P: 1750837942185812 usec, L: 0
0625 07:52:32.959355 (+ 122us) write_op.cc:270] Start()
0625 07:52:32.959386 (+ 31us) write_op.cc:276] Timestamp: P: 1750837942186633 usec, L: 0
0625 07:52:32.959523 (+ 137us) write_op.cc:270] Start()
0625 07:52:32.959561 (+ 38us) write_op.cc:276] Timestamp: P: 1750837942329871 usec, L: 0
0625 07:52:32.959675 (+ 114us) write_op.cc:270] Start()
0625 07:52:32.959693 (+ 18us) write_op.cc:276] Timestamp: P: 1750837942336050 usec, L: 0
0625 07:52:32.959848 (+ 155us) write_op.cc:270] Start()
0625 07:52:32.959868 (+ 20us) write_op.cc:276] Timestamp: P: 1750837942495547 usec, L: 0
0625 07:52:32.960006 (+ 138us) write_op.cc:270] Start()
0625 07:52:32.960023 (+ 17us) write_op.cc:276] Timestamp: P: 1750837942496159 usec, L: 0
0625 07:52:32.960147 (+ 124us) write_op.cc:270] Start()
0625 07:52:32.960162 (+ 15us) write_op.cc:276] Timestamp: P: 1750837942500797 usec, L: 0
0625 07:52:32.960293 (+ 131us) write_op.cc:270] Start()
0625 07:52:32.960322 (+ 29us) write_op.cc:276] Timestamp: P: 1750837942501318 usec, L: 0
0625 07:52:32.960464 (+ 142us) write_op.cc:270] Start()
0625 07:52:32.960486 (+ 22us) write_op.cc:276] Timestamp: P: 1750837942504805 usec, L: 0
0625 07:52:32.960663 (+ 177us) write_op.cc:270] Start()
0625 07:52:32.960703 (+ 40us) write_op.cc:276] Timestamp: P: 1750837942633120 usec, L: 0
0625 07:52:32.960856 (+ 153us) write_op.cc:270] Start()
0625 07:52:32.960881 (+ 25us) write_op.cc:276] Timestamp: P: 1750837942634259 usec, L: 0
0625 07:52:32.982441 (+ 21560us) write_op.cc:270] Start()
0625 07:52:32.982469 (+ 28us) write_op.cc:276] Timestamp: P: 1750837942635116 usec, L: 0
0625 07:52:32.982716 (+ 247us) write_op.cc:270] Start()
0625 07:52:32.982746 (+ 30us) write_op.cc:276] Timestamp: P: 1750837942635969 usec, L: 0
0625 07:52:32.982917 (+ 171us) write_op.cc:270] Start()
0625 07:52:32.982942 (+ 25us) write_op.cc:276] Timestamp: P: 1750837942636828 usec, L: 0
0625 07:52:32.983088 (+ 146us) write_op.cc:270] Start()
0625 07:52:32.983112 (+ 24us) write_op.cc:276] Timestamp: P: 1750837942646607 usec, L: 0
0625 07:52:32.983256 (+ 144us) write_op.cc:270] Start()
0625 07:52:32.983280 (+ 24us) write_op.cc:276] Timestamp: P: 1750837942727777 usec, L: 0
0625 07:52:32.983510 (+ 230us) write_op.cc:270] Start()
0625 07:52:32.983548 (+ 38us) write_op.cc:276] Timestamp: P: 1750837942731309 usec, L: 0
0625 07:52:32.983732 (+ 184us) write_op.cc:270] Start()
0625 07:52:32.983756 (+ 24us) write_op.cc:276] Timestamp: P: 1750837942739665 usec, L: 0
0625 07:52:32.983914 (+ 158us) write_op.cc:270] Start()
0625 07:52:32.983939 (+ 25us) write_op.cc:276] Timestamp: P: 1750837942778981 usec, L: 0
0625 07:52:32.984081 (+ 142us) write_op.cc:270] Start()
0625 07:52:32.984117 (+ 36us) write_op.cc:276] Timestamp: P: 1750837942790892 usec, L: 0
0625 07:52:32.984270 (+ 153us) write_op.cc:270] Start()
0625 07:52:32.984294 (+ 24us) write_op.cc:276] Timestamp: P: 1750837942798839 usec, L: 0
0625 07:52:32.984442 (+ 148us) write_op.cc:270] Start()
0625 07:52:32.984465 (+ 23us) write_op.cc:276] Timestamp: P: 1750837942846548 usec, L: 0
0625 07:52:32.984642 (+ 177us) write_op.cc:270] Start()
0625 07:52:32.984665 (+ 23us) write_op.cc:276] Timestamp: P: 1750837942855182 usec, L: 0
0625 07:52:33.033619 (+ 48954us) write_op.cc:270] Start()
0625 07:52:33.033661 (+ 42us) write_op.cc:276] Timestamp: P: 1750837942855881 usec, L: 0
0625 07:52:33.033881 (+ 220us) write_op.cc:270] Start()
0625 07:52:33.033905 (+ 24us) write_op.cc:276] Timestamp: P: 1750837943032407 usec, L: 0
0625 07:52:33.034088 (+ 183us) write_op.cc:270] Start()
0625 07:52:33.034111 (+ 23us) write_op.cc:276] Timestamp: P: 1750837943033973 usec, L: 0
0625 07:52:33.034260 (+ 149us) write_op.cc:270] Start()
0625 07:52:33.034278 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943035110 usec, L: 0
0625 07:52:33.034387 (+ 109us) write_op.cc:270] Start()
0625 07:52:33.034404 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943036227 usec, L: 0
0625 07:52:33.034522 (+ 118us) write_op.cc:270] Start()
0625 07:52:33.034551 (+ 29us) write_op.cc:276] Timestamp: P: 1750837943047196 usec, L: 0
0625 07:52:33.034664 (+ 113us) write_op.cc:270] Start()
0625 07:52:33.034680 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943049057 usec, L: 0
0625 07:52:33.034807 (+ 127us) write_op.cc:270] Start()
0625 07:52:33.034826 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943050175 usec, L: 0
0625 07:52:33.034955 (+ 129us) write_op.cc:270] Start()
0625 07:52:33.034978 (+ 23us) write_op.cc:276] Timestamp: P: 1750837943052630 usec, L: 0
0625 07:52:33.035095 (+ 117us) write_op.cc:270] Start()
0625 07:52:33.035112 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943054334 usec, L: 0
0625 07:52:33.035219 (+ 107us) write_op.cc:270] Start()
0625 07:52:33.035235 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943055695 usec, L: 0
0625 07:52:33.035366 (+ 131us) write_op.cc:270] Start()
0625 07:52:33.035383 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943057452 usec, L: 0
0625 07:52:33.035490 (+ 107us) write_op.cc:270] Start()
0625 07:52:33.035507 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943059056 usec, L: 0
0625 07:52:33.035641 (+ 134us) write_op.cc:270] Start()
0625 07:52:33.035659 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943076176 usec, L: 0
0625 07:52:33.035773 (+ 114us) write_op.cc:270] Start()
0625 07:52:33.035791 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943076917 usec, L: 0
0625 07:52:33.035911 (+ 120us) write_op.cc:270] Start()
0625 07:52:33.035925 (+ 14us) write_op.cc:276] Timestamp: P: 1750837943077509 usec, L: 0
0625 07:52:33.036051 (+ 126us) write_op.cc:270] Start()
0625 07:52:33.036067 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943078118 usec, L: 0
0625 07:52:33.036214 (+ 147us) write_op.cc:270] Start()
0625 07:52:33.036238 (+ 24us) write_op.cc:276] Timestamp: P: 1750837943083268 usec, L: 0
0625 07:52:33.036390 (+ 152us) write_op.cc:270] Start()
0625 07:52:33.036408 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943101284 usec, L: 0
0625 07:52:33.036541 (+ 133us) write_op.cc:270] Start()
0625 07:52:33.036560 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943101887 usec, L: 0
0625 07:52:33.036683 (+ 123us) write_op.cc:270] Start()
0625 07:52:33.229655 (+192972us) write_op.cc:276] Timestamp: P: 1750837943102407 usec, L: 0
0625 07:52:33.229904 (+ 249us) write_op.cc:270] Start()
0625 07:52:33.229993 (+ 89us) write_op.cc:276] Timestamp: P: 1750837943125504 usec, L: 0
0625 07:52:33.230719 (+ 726us) write_op.cc:270] Start()
0625 07:52:33.230750 (+ 31us) write_op.cc:276] Timestamp: P: 1750837943127061 usec, L: 0
0625 07:52:33.230907 (+ 157us) write_op.cc:270] Start()
0625 07:52:33.230926 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943130124 usec, L: 0
0625 07:52:33.231042 (+ 116us) write_op.cc:270] Start()
0625 07:52:33.231059 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943236000 usec, L: 0
0625 07:52:33.231171 (+ 112us) write_op.cc:270] Start()
0625 07:52:33.231186 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943275328 usec, L: 0
0625 07:52:33.231281 (+ 95us) write_op.cc:270] Start()
0625 07:52:33.231296 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943276051 usec, L: 0
0625 07:52:33.231406 (+ 110us) write_op.cc:270] Start()
0625 07:52:33.231421 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943276759 usec, L: 0
0625 07:52:33.231567 (+ 146us) write_op.cc:270] Start()
0625 07:52:33.231584 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943277380 usec, L: 0
0625 07:52:33.231695 (+ 111us) write_op.cc:270] Start()
0625 07:52:33.231711 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943277970 usec, L: 0
0625 07:52:33.231822 (+ 111us) write_op.cc:270] Start()
0625 07:52:33.231837 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943281119 usec, L: 0
0625 07:52:33.231935 (+ 98us) write_op.cc:270] Start()
0625 07:52:33.231950 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943305072 usec, L: 0
0625 07:52:33.232052 (+ 102us) write_op.cc:270] Start()
0625 07:52:33.232067 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943316169 usec, L: 0
0625 07:52:33.232192 (+ 125us) write_op.cc:270] Start()
0625 07:52:33.232208 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943319947 usec, L: 0
0625 07:52:33.232346 (+ 138us) write_op.cc:270] Start()
0625 07:52:33.232362 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943333408 usec, L: 0
0625 07:52:33.232467 (+ 105us) write_op.cc:270] Start()
0625 07:52:33.232482 (+ 15us) write_op.cc:276] Timestamp: P: 1750837943370317 usec, L: 0
0625 07:52:33.232598 (+ 116us) write_op.cc:270] Start()
0625 07:52:33.232614 (+ 16us) write_op.cc:276] Timestamp: P: 1750837943373107 usec, L: 0
0625 07:52:33.262260 (+ 29646us) write_op.cc:270] Start()
0625 07:52:33.262288 (+ 28us) write_op.cc:276] Timestamp: P: 1750837943378717 usec, L: 0
0625 07:52:33.262495 (+ 207us) write_op.cc:270] Start()
0625 07:52:33.262520 (+ 25us) write_op.cc:276] Timestamp: P: 1750837943380976 usec, L: 0
0625 07:52:33.262696 (+ 176us) write_op.cc:270] Start()
0625 07:52:33.262717 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943383334 usec, L: 0
0625 07:52:33.262865 (+ 148us) write_op.cc:270] Start()
0625 07:52:33.262884 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943398810 usec, L: 0
0625 07:52:33.263012 (+ 128us) write_op.cc:270] Start()
0625 07:52:33.263029 (+ 17us) write_op.cc:276] Timestamp: P: 1750837943400736 usec, L: 0
0625 07:52:33.263143 (+ 114us) write_op.cc:270] Start()
0625 07:52:33.263161 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943401340 usec, L: 0
0625 07:52:33.263319 (+ 158us) write_op.cc:270] Start()
0625 07:52:33.263340 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943401969 usec, L: 0
0625 07:52:33.263475 (+ 135us) write_op.cc:270] Start()
0625 07:52:33.263495 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943515646 usec, L: 0
0625 07:52:33.263638 (+ 143us) write_op.cc:270] Start()
0625 07:52:33.263657 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943526321 usec, L: 0
0625 07:52:33.263838 (+ 181us) write_op.cc:270] Start()
0625 07:52:33.263860 (+ 22us) write_op.cc:276] Timestamp: P: 1750837943527981 usec, L: 0
0625 07:52:33.264019 (+ 159us) write_op.cc:270] Start()
0625 07:52:33.264041 (+ 22us) write_op.cc:276] Timestamp: P: 1750837943528650 usec, L: 0
0625 07:52:33.264202 (+ 161us) write_op.cc:270] Start()
0625 07:52:33.264225 (+ 23us) write_op.cc:276] Timestamp: P: 1750837943546612 usec, L: 0
0625 07:52:33.264357 (+ 132us) write_op.cc:270] Start()
0625 07:52:33.264377 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943577152 usec, L: 0
0625 07:52:33.264502 (+ 125us) write_op.cc:270] Start()
0625 07:52:33.264520 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943579595 usec, L: 0
0625 07:52:33.277826 (+ 13306us) write_op.cc:270] Start()
0625 07:52:33.277857 (+ 31us) write_op.cc:276] Timestamp: P: 1750837943580347 usec, L: 0
0625 07:52:33.278075 (+ 218us) write_op.cc:270] Start()
0625 07:52:33.278098 (+ 23us) write_op.cc:276] Timestamp: P: 1750837943582625 usec, L: 0
0625 07:52:33.278228 (+ 130us) write_op.cc:270] Start()
0625 07:52:33.278248 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943583508 usec, L: 0
0625 07:52:33.278400 (+ 152us) write_op.cc:270] Start()
0625 07:52:33.278420 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943594135 usec, L: 0
0625 07:52:33.278544 (+ 124us) write_op.cc:270] Start()
0625 07:52:33.278564 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943597577 usec, L: 0
0625 07:52:33.278678 (+ 114us) write_op.cc:270] Start()
0625 07:52:33.278696 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943601555 usec, L: 0
0625 07:52:33.278900 (+ 204us) write_op.cc:270] Start()
0625 07:52:33.278920 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943602278 usec, L: 0
0625 07:52:33.279148 (+ 228us) write_op.cc:270] Start()
0625 07:52:33.279168 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943603202 usec, L: 0
0625 07:52:33.279333 (+ 165us) write_op.cc:270] Start()
0625 07:52:33.279357 (+ 24us) write_op.cc:276] Timestamp: P: 1750837943603954 usec, L: 0
0625 07:52:33.279509 (+ 152us) write_op.cc:270] Start()
0625 07:52:33.279530 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943608923 usec, L: 0
0625 07:52:33.286907 (+ 7377us) write_op.cc:270] Start()
0625 07:52:33.286932 (+ 25us) write_op.cc:276] Timestamp: P: 1750837943609675 usec, L: 0
0625 07:52:33.287089 (+ 157us) write_op.cc:270] Start()
0625 07:52:33.287108 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943610488 usec, L: 0
0625 07:52:33.287237 (+ 129us) write_op.cc:270] Start()
0625 07:52:33.287255 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943611142 usec, L: 0
0625 07:52:33.287385 (+ 130us) write_op.cc:270] Start()
0625 07:52:33.287403 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943688441 usec, L: 0
0625 07:52:33.287557 (+ 154us) write_op.cc:270] Start()
0625 07:52:33.302110 (+ 14553us) write_op.cc:276] Timestamp: P: 1750837943704467 usec, L: 0
0625 07:52:33.302449 (+ 339us) write_op.cc:270] Start()
0625 07:52:33.302478 (+ 29us) write_op.cc:276] Timestamp: P: 1750837943706668 usec, L: 0
0625 07:52:33.302649 (+ 171us) write_op.cc:270] Start()
0625 07:52:33.302674 (+ 25us) write_op.cc:276] Timestamp: P: 1750837943707962 usec, L: 0
0625 07:52:33.302849 (+ 175us) write_op.cc:270] Start()
0625 07:52:33.302870 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943710827 usec, L: 0
0625 07:52:33.305661 (+ 2791us) write_op.cc:270] Start()
0625 07:52:33.305711 (+ 50us) write_op.cc:276] Timestamp: P: 1750837943744290 usec, L: 0
0625 07:52:33.305891 (+ 180us) write_op.cc:270] Start()
0625 07:52:33.305912 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943747186 usec, L: 0
0625 07:52:33.306070 (+ 158us) write_op.cc:270] Start()
0625 07:52:33.306090 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943802963 usec, L: 0
0625 07:52:33.306219 (+ 129us) write_op.cc:270] Start()
0625 07:52:33.306237 (+ 18us) write_op.cc:276] Timestamp: P: 1750837943804636 usec, L: 0
0625 07:52:33.306365 (+ 128us) write_op.cc:270] Start()
0625 07:52:33.306384 (+ 19us) write_op.cc:276] Timestamp: P: 1750837943854388 usec, L: 0
0625 07:52:33.306508 (+ 124us) write_op.cc:270] Start()
0625 07:52:33.306529 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943855639 usec, L: 0
0625 07:52:33.306710 (+ 181us) write_op.cc:270] Start()
0625 07:52:33.306747 (+ 37us) write_op.cc:276] Timestamp: P: 1750837943857362 usec, L: 0
0625 07:52:33.306889 (+ 142us) write_op.cc:270] Start()
0625 07:52:33.306909 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943868149 usec, L: 0
0625 07:52:33.307043 (+ 134us) write_op.cc:270] Start()
0625 07:52:33.307063 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943870801 usec, L: 0
0625 07:52:33.307225 (+ 162us) write_op.cc:270] Start()
0625 07:52:33.307246 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943914904 usec, L: 0
0625 07:52:33.307411 (+ 165us) write_op.cc:270] Start()
0625 07:52:33.307432 (+ 21us) write_op.cc:276] Timestamp: P: 1750837943920795 usec, L: 0
0625 07:52:33.307611 (+ 179us) write_op.cc:270] Start()
0625 07:52:33.307631 (+ 20us) write_op.cc:276] Timestamp: P: 1750837943921833 usec, L: 0
0625 07:52:33.307773 (+ 142us) write_op.cc:270] Start()
0625 07:52:33.307802 (+ 29us) write_op.cc:276] Timestamp: P: 1750837943922445 usec, L: 0
0625 07:52:33.307925 (+ 123us) write_op.cc:270] Start()
0625 07:52:33.307944 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944018944 usec, L: 0
0625 07:52:33.308083 (+ 139us) write_op.cc:270] Start()
0625 07:52:33.308106 (+ 23us) write_op.cc:276] Timestamp: P: 1750837944019617 usec, L: 0
0625 07:52:33.308259 (+ 153us) write_op.cc:270] Start()
0625 07:52:33.308279 (+ 20us) write_op.cc:276] Timestamp: P: 1750837944026273 usec, L: 0
0625 07:52:33.308415 (+ 136us) write_op.cc:270] Start()
0625 07:52:33.308439 (+ 24us) write_op.cc:276] Timestamp: P: 1750837944026849 usec, L: 0
0625 07:52:33.308621 (+ 182us) write_op.cc:270] Start()
0625 07:52:33.308644 (+ 23us) write_op.cc:276] Timestamp: P: 1750837944027480 usec, L: 0
0625 07:52:33.337047 (+ 28403us) write_op.cc:270] Start()
0625 07:52:33.337085 (+ 38us) write_op.cc:276] Timestamp: P: 1750837944047572 usec, L: 0
0625 07:52:33.337265 (+ 180us) write_op.cc:270] Start()
0625 07:52:33.337285 (+ 20us) write_op.cc:276] Timestamp: P: 1750837944059362 usec, L: 0
0625 07:52:33.350093 (+ 12808us) write_op.cc:270] Start()
0625 07:52:33.350124 (+ 31us) write_op.cc:276] Timestamp: P: 1750837944060845 usec, L: 0
0625 07:52:33.350264 (+ 140us) write_op.cc:270] Start()
0625 07:52:33.350282 (+ 18us) write_op.cc:276] Timestamp: P: 1750837944061786 usec, L: 0
0625 07:52:33.350401 (+ 119us) write_op.cc:270] Start()
0625 07:52:33.350422 (+ 21us) write_op.cc:276] Timestamp: P: 1750837944062676 usec, L: 0
0625 07:52:33.350586 (+ 164us) write_op.cc:270] Start()
0625 07:52:33.350625 (+ 39us) write_op.cc:276] Timestamp: P: 1750837944069932 usec, L: 0
0625 07:52:33.350786 (+ 161us) write_op.cc:270] Start()
0625 07:52:33.350814 (+ 28us) write_op.cc:276] Timestamp: P: 1750837944073911 usec, L: 0
0625 07:52:33.350950 (+ 136us) write_op.cc:270] Start()
0625 07:52:33.350969 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944074950 usec, L: 0
0625 07:52:33.351094 (+ 125us) write_op.cc:270] Start()
0625 07:52:33.351113 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944119917 usec, L: 0
0625 07:52:33.351250 (+ 137us) write_op.cc:270] Start()
0625 07:52:33.351271 (+ 21us) write_op.cc:276] Timestamp: P: 1750837944124175 usec, L: 0
0625 07:52:33.351426 (+ 155us) write_op.cc:270] Start()
0625 07:52:33.361904 (+ 10478us) write_op.cc:276] Timestamp: P: 1750837944148928 usec, L: 0
0625 07:52:33.362115 (+ 211us) write_op.cc:270] Start()
0625 07:52:33.362136 (+ 21us) write_op.cc:276] Timestamp: P: 1750837944225839 usec, L: 0
0625 07:52:33.362277 (+ 141us) write_op.cc:270] Start()
0625 07:52:33.362295 (+ 18us) write_op.cc:276] Timestamp: P: 1750837944226543 usec, L: 0
0625 07:52:33.362416 (+ 121us) write_op.cc:270] Start()
0625 07:52:33.362433 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944239462 usec, L: 0
0625 07:52:33.362537 (+ 104us) write_op.cc:270] Start()
0625 07:52:33.362554 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944240172 usec, L: 0
0625 07:52:33.362661 (+ 107us) write_op.cc:270] Start()
0625 07:52:33.362677 (+ 16us) write_op.cc:276] Timestamp: P: 1750837944304559 usec, L: 0
0625 07:52:33.362806 (+ 129us) write_op.cc:270] Start()
0625 07:52:33.362823 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944306138 usec, L: 0
0625 07:52:33.362924 (+ 101us) write_op.cc:270] Start()
0625 07:52:33.362939 (+ 15us) write_op.cc:276] Timestamp: P: 1750837944307142 usec, L: 0
0625 07:52:33.368884 (+ 5945us) write_op.cc:270] Start()
0625 07:52:33.368910 (+ 26us) write_op.cc:276] Timestamp: P: 1750837944308427 usec, L: 0
0625 07:52:33.369077 (+ 167us) write_op.cc:270] Start()
0625 07:52:33.369102 (+ 25us) write_op.cc:276] Timestamp: P: 1750837944309304 usec, L: 0
0625 07:52:33.369239 (+ 137us) write_op.cc:270] Start()
0625 07:52:33.369258 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944310392 usec, L: 0
0625 07:52:33.369380 (+ 122us) write_op.cc:270] Start()
0625 07:52:33.369399 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944311925 usec, L: 0
0625 07:52:33.369543 (+ 144us) write_op.cc:270] Start()
0625 07:52:33.369563 (+ 20us) write_op.cc:276] Timestamp: P: 1750837944445146 usec, L: 0
0625 07:52:33.369692 (+ 129us) write_op.cc:270] Start()
0625 07:52:33.369709 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944453036 usec, L: 0
0625 07:52:33.369836 (+ 127us) write_op.cc:270] Start()
0625 07:52:33.369853 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944476215 usec, L: 0
0625 07:52:33.369975 (+ 122us) write_op.cc:270] Start()
0625 07:52:33.369990 (+ 15us) write_op.cc:276] Timestamp: P: 1750837944477206 usec, L: 0
0625 07:52:33.370095 (+ 105us) write_op.cc:270] Start()
0625 07:52:33.370112 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944479166 usec, L: 0
0625 07:52:33.370217 (+ 105us) write_op.cc:270] Start()
0625 07:52:33.370232 (+ 15us) write_op.cc:276] Timestamp: P: 1750837944480133 usec, L: 0
0625 07:52:33.370335 (+ 103us) write_op.cc:270] Start()
0625 07:52:33.370350 (+ 15us) write_op.cc:276] Timestamp: P: 1750837944481039 usec, L: 0
0625 07:52:33.370459 (+ 109us) write_op.cc:270] Start()
0625 07:52:33.370475 (+ 16us) write_op.cc:276] Timestamp: P: 1750837944481936 usec, L: 0
0625 07:52:33.370581 (+ 106us) write_op.cc:270] Start()
0625 07:52:33.370598 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944482720 usec, L: 0
0625 07:52:33.370694 (+ 96us) write_op.cc:270] Start()
0625 07:52:33.370709 (+ 15us) write_op.cc:276] Timestamp: P: 1750837944483538 usec, L: 0
0625 07:52:33.370824 (+ 115us) write_op.cc:270] Start()
0625 07:52:33.370841 (+ 17us) write_op.cc:276] Timestamp: P: 1750837944553613 usec, L: 0
0625 07:52:33.370949 (+ 108us) write_op.cc:270] Start()
0625 07:52:33.370965 (+ 16us) write_op.cc:276] Timestamp: P: 1750837944556120 usec, L: 0
0625 07:52:33.371142 (+ 177us) write_op.cc:270] Start()
0625 07:52:33.371162 (+ 20us) write_op.cc:276] Timestamp: P: 1750837944629227 usec, L: 0
0625 07:52:33.371296 (+ 134us) write_op.cc:270] Start()
0625 07:52:33.371315 (+ 19us) write_op.cc:276] Timestamp: P: 1750837944721752 usec, L: 0
0625 07:52:33.371424 (+ 109us) write_op.cc:270] Start()
0625 07:52:33.371440 (+ 16u
I20250625 07:52:35.202831 2348 raft_consensus.cc:513] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.203963 1161 tablet_replica.cc:331] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.204167 1161 raft_consensus.cc:2241] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.204329 1161 raft_consensus.cc:2270] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.205407 2178 raft_consensus.cc:491] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.205579 2178 raft_consensus.cc:513] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.205644 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:35.206094 713 leader_election.cc:343] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.206218 2178 leader_election.cc:290] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:35.206606 2348 leader_election.cc:290] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.207051 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.207237 914 raft_consensus.cc:3058] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:35.207369 986 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:35.209889 914 raft_consensus.cc:2466] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:35.210292 985 leader_election.cc:343] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:35.210409 985 leader_election.cc:304] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.210847 2178 raft_consensus.cc:2747] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.211179 713 leader_election.cc:304] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.211643 2348 raft_consensus.cc:2802] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.211817 2348 raft_consensus.cc:695] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.212150 2348 consensus_queue.cc:237] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.215921 1163 tablet_bootstrap.cc:654] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.216907 2178 raft_consensus.cc:491] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.217060 2178 raft_consensus.cc:513] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.217615 2178 leader_election.cc:290] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:35.218485 986 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: be7b755d95b14ecea7a691003455f381
W20250625 07:52:35.219280 985 leader_election.cc:343] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: be7b755d95b14ecea7a691003455f381
I20250625 07:52:35.219403 985 leader_election.cc:304] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.219813 2178 raft_consensus.cc:2747] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.224651 2366 tablet.cc:1722] T 1405e55703324bc7a1e26fe2c349bda7 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.205432 1161 ts_tablet_manager.cc:1905] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.229748 2367 tablet.cc:1722] T 8cadd41655404316b10939e46d370c76 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.232216 1158 tablet_bootstrap.cc:492] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.232404 1158 ts_tablet_manager.cc:1397] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250625 07:52:35.233139 1158 raft_consensus.cc:357] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.233341 1158 raft_consensus.cc:383] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.233440 1158 raft_consensus.cc:738] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.233711 1158 consensus_queue.cc:260] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.234284 1158 ts_tablet_manager.cc:1428] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.234648 1158 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.235246 2365 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:35.236491 1158 tablet_bootstrap.cc:654] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.237462 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_88b19fcde2d84c5e960e47a65b073f24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.238027 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88b19fcde2d84c5e960e47a65b073f24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.241802 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:35.241991 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:35.242233 2340 tablet.cc:1722] T cffde34fbba34e44b67985d3480666c3 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.256451 1162 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.256669 1162 ts_tablet_manager.cc:1397] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.002s
I20250625 07:52:35.257512 1162 raft_consensus.cc:357] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.257741 1162 raft_consensus.cc:383] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.257843 1162 raft_consensus.cc:738] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.262902 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4393f5cdd7ea48e6ae82d10d76875820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.263422 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4393f5cdd7ea48e6ae82d10d76875820 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.269429 1158 tablet_bootstrap.cc:492] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.269632 1158 ts_tablet_manager.cc:1397] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250625 07:52:35.258082 1162 consensus_queue.cc:260] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.270004 2365 tablet.cc:1722] T cffde34fbba34e44b67985d3480666c3 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.270582 1162 ts_tablet_manager.cc:1428] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20250625 07:52:35.270763 1158 raft_consensus.cc:357] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.270989 1158 raft_consensus.cc:383] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.271104 1158 raft_consensus.cc:738] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.271392 1158 consensus_queue.cc:260] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.272080 1162 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.272413 1158 ts_tablet_manager.cc:1428] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250625 07:52:35.272775 1158 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.274504 1162 tablet_bootstrap.cc:654] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.277537 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_250afd2ed99c43b4a24721d178e5e372 [id=29e4ffaed4f148b6b10e38231435c071] alter complete (version 2)
I20250625 07:52:35.278133 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_57546ca5ffb04419aaa4086e381744e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.278667 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57546ca5ffb04419aaa4086e381744e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.279243 1161 ts_tablet_manager.cc:1918] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.279397 1161 log.cc:1199] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/d76681f162074b67a6b886887b5a3d61
I20250625 07:52:35.279816 1161 ts_tablet_manager.cc:1939] T d76681f162074b67a6b886887b5a3d61 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.280212 2389 tablet.cc:1722] T c934a46a8f6c407caa7a84d6415d6901 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.280681 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1170) took 1139 ms. Trace:
I20250625 07:52:35.290099 1161 rpcz_store.cc:276] 0625 07:52:34.141178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.153545 (+ 12367us) service_pool.cc:225] Handling call
0625 07:52:35.203900 (+1050355us) ts_tablet_manager.cc:1145] Deleting tablet d76681f162074b67a6b886887b5a3d61
0625 07:52:35.203910 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.239375 (+ 35465us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.279114 (+ 39739us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.280602 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050131}
I20250625 07:52:35.290498 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet d76681f162074b67a6b886887b5a3d61 (table default.Table_96d96b9fee4d48efaeb2f22ff67b43e6 [id=8340c9a7e5874bf486d4f07192ebb908]) successfully deleted
I20250625 07:52:35.292755 1161 tablet_replica.cc:331] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.292964 1161 raft_consensus.cc:2241] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.294751 2365 tablet.cc:1722] T 5de1492e483e47919ee5581ecee227be P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:35.281322 2370 tablet.cc:1722] T cffde34fbba34e44b67985d3480666c3 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.276021 2161 raft_consensus.cc:491] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.295416 2161 raft_consensus.cc:513] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.296000 2161 leader_election.cc:290] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.288388 2346 tablet.cc:1722] T c934a46a8f6c407caa7a84d6415d6901 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.296541 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2e441da0a6b34a83a84efaa1c2041ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.297101 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e441da0a6b34a83a84efaa1c2041ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.300549 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0657163b9a0d4f7f99f57f7fd1f32462" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.300601 713 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0657163b9a0d4f7f99f57f7fd1f32462
W20250625 07:52:35.301107 713 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.301220 713 leader_election.cc:304] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.301640 1161 raft_consensus.cc:2270] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.302203 1161 ts_tablet_manager.cc:1905] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.302732 2161 raft_consensus.cc:2747] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.308446 1158 tablet_bootstrap.cc:654] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.307925 2346 tablet.cc:1722] T e109c9dca7444681ac517c7fc3e14dfa P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 07:52:35.314462 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b47065e0382b4b9a936613a0ad84695d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b47065e0382b4b9a936613a0ad84695d already in progress: creating tablet
W20250625 07:52:35.315310 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 168f1a13533045d89ba1f07a72d4ff63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168f1a13533045d89ba1f07a72d4ff63 already in progress: creating tablet
I20250625 07:52:35.315704 1160 tablet_replica.cc:331] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:35.315958 1160 raft_consensus.cc:2241] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.316191 1160 raft_consensus.cc:2270] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.316773 1160 ts_tablet_manager.cc:1905] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.319751 2389 tablet.cc:1722] T e109c9dca7444681ac517c7fc3e14dfa P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:35.319715 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d5b8fff6c167473b8b81e9a4e15f4f32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.320241 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b8fff6c167473b8b81e9a4e15f4f32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.322301 1163 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.322468 1163 ts_tablet_manager.cc:1397] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.149s user 0.001s sys 0.002s
I20250625 07:52:35.323262 1163 raft_consensus.cc:357] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.323444 1163 raft_consensus.cc:383] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.323518 1163 raft_consensus.cc:738] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.323746 1163 consensus_queue.cc:260] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.324551 1163 ts_tablet_manager.cc:1428] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:35.324589 2311 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250625 07:52:35.325054 1163 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.326087 1161 ts_tablet_manager.cc:1918] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.326205 1161 log.cc:1199] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/06e4bf78218f4b82b4f16f59b1704256
I20250625 07:52:35.326507 1161 ts_tablet_manager.cc:1939] T 06e4bf78218f4b82b4f16f59b1704256 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.326782 1163 tablet_bootstrap.cc:654] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.340873 1158 tablet_bootstrap.cc:492] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.341091 1162 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.341130 1158 ts_tablet_manager.cc:1397] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20250625 07:52:35.341327 1162 ts_tablet_manager.cc:1397] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.003s
I20250625 07:52:35.341886 1158 raft_consensus.cc:357] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.342085 1158 raft_consensus.cc:383] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.342164 1158 raft_consensus.cc:738] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.342137 1162 raft_consensus.cc:357] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.342455 1162 raft_consensus.cc:383] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.342427 1158 consensus_queue.cc:260] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.342550 1162 raft_consensus.cc:738] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.342974 1162 consensus_queue.cc:260] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.343505 1162 ts_tablet_manager.cc:1428] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:35.349253 1158 ts_tablet_manager.cc:1428] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:35.349542 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1169) took 1208 ms. Trace:
I20250625 07:52:35.349673 1161 rpcz_store.cc:276] 0625 07:52:34.141071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.168267 (+ 27196us) service_pool.cc:225] Handling call
0625 07:52:35.292642 (+1124375us) ts_tablet_manager.cc:1145] Deleting tablet 06e4bf78218f4b82b4f16f59b1704256
0625 07:52:35.292652 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.319658 (+ 27006us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.325986 (+ 6328us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.349506 (+ 23520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115803}
I20250625 07:52:35.350020 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 06e4bf78218f4b82b4f16f59b1704256 (table default.Table_4f95753979a34ea994409ba5978b7186 [id=4ff84ce52a874a5792b79f35f2016f1b]) successfully deleted
I20250625 07:52:35.351442 1162 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.353243 1162 tablet_bootstrap.cc:654] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.354017 1158 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.355757 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1176) took 1033 ms. Trace:
I20250625 07:52:35.355880 1161 rpcz_store.cc:276] 0625 07:52:34.322535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.322592 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:35.355621 (+1033029us) ts_tablet_manager.cc:1145] Deleting tablet b47065e0382b4b9a936613a0ad84695d
0625 07:52:35.355631 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.355726 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025036}
W20250625 07:52:35.356231 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b47065e0382b4b9a936613a0ad84695d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b47065e0382b4b9a936613a0ad84695d already in progress: creating tablet
I20250625 07:52:35.358155 1161 tablet_replica.cc:331] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.358371 1161 raft_consensus.cc:2241] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.358562 1161 raft_consensus.cc:2270] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.359087 1161 ts_tablet_manager.cc:1905] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.359723 1158 tablet_bootstrap.cc:654] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.363824 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_146999741edf4bde91a875f03e40a74e" } new_table_name: "default.Table_c79988fc838d4a2f9aac01a3e460397d" modify_external_catalogs: true
I20250625 07:52:35.365120 2045 raft_consensus.cc:491] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.365339 2045 raft_consensus.cc:513] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.366086 2045 leader_election.cc:290] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.366715 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.366724 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.366916 908 raft_consensus.cc:2466] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:35.367476 713 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.367667 713 leader_election.cc:304] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.367924 2045 raft_consensus.cc:2802] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.368002 2045 raft_consensus.cc:491] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.368094 2045 raft_consensus.cc:3058] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.368458 1162 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.372007 1162 ts_tablet_manager.cc:1397] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250625 07:52:35.372288 2112 raft_consensus.cc:491] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.372427 2112 raft_consensus.cc:513] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.373060 2112 leader_election.cc:290] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.373042 1162 raft_consensus.cc:357] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.373200 1162 raft_consensus.cc:383] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.373250 1162 raft_consensus.cc:738] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.373446 1162 consensus_queue.cc:260] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.373476 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bc8f47541c54c4287ebb7b892daa1be" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.373591 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bc8f47541c54c4287ebb7b892daa1be" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.373915 1162 ts_tablet_manager.cc:1428] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250625 07:52:35.373982 850 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.374155 1162 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:35.374156 851 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.374276 851 leader_election.cc:304] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.374828 2112 raft_consensus.cc:2747] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.377895 1048 raft_consensus.cc:3058] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:35.378002 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f297fdcad1449c1a4cb6a5984bb5a9c consensus state has no leader
I20250625 07:52:35.384831 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_ec11ec9d97c040abb492fec0547ba805 [id=23322aa9321140c39272b5b2a102fb02] alter complete (version 5)
I20250625 07:52:35.385692 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 743123ffcf8240dc82bea7178559b7ea with delete_type TABLET_DATA_DELETED (replaced by tablet 90f2bef7baa54f1a96b3b59921d59457) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.385823 1160 ts_tablet_manager.cc:1918] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.385954 1160 log.cc:1199] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/da895c90d6ee40379fe8181c2dbd7a1e
I20250625 07:52:35.386179 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 1afe90251c964e369d05e82c7db19326 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.386281 1160 ts_tablet_manager.cc:1939] T da895c90d6ee40379fe8181c2dbd7a1e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:35.386479 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ab38e0cdcd14f209e6877f2e4a46e98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:15 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.387025 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1208) took 1039 ms. Trace:
I20250625 07:52:35.387146 1160 rpcz_store.cc:276] 0625 07:52:34.347901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.348150 (+ 249us) service_pool.cc:225] Handling call
0625 07:52:35.315638 (+967488us) ts_tablet_manager.cc:1145] Deleting tablet da895c90d6ee40379fe8181c2dbd7a1e
0625 07:52:35.315647 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.362365 (+ 46718us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.385679 (+ 23314us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.386982 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":967302}
I20250625 07:52:35.387542 1160 tablet_replica.cc:331] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:35.387738 1160 raft_consensus.cc:2241] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.387887 1160 raft_consensus.cc:2270] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.370841 2045 raft_consensus.cc:513] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.388415 1160 ts_tablet_manager.cc:1905] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.388880 2045 leader_election.cc:290] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.389343 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 53648.1us
I20250625 07:52:35.389423 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.389606 908 raft_consensus.cc:3058] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.389724 2178 raft_consensus.cc:491] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.389866 2178 raft_consensus.cc:513] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.390401 2178 leader_election.cc:290] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:35.391714 986 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:35.391790 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cc1842f257d409ba0a56873f582f023" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:35.392050 985 leader_election.cc:343] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:35.392150 985 leader_election.cc:304] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.392378 908 raft_consensus.cc:2466] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.392838 713 leader_election.cc:304] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:35.393229 713 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.393887 2178 raft_consensus.cc:2747] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.393851 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_47d9a728e38f49cf948b3bdf5ff09621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:35.396171 2045 raft_consensus.cc:2802] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.396400 2045 raft_consensus.cc:695] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.396813 2045 consensus_queue.cc:237] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.398878 1162 tablet_bootstrap.cc:654] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.403328 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6db07140365b4f2a8c6d6aaaf164be23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:35.405670 1163 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.405977 1163 ts_tablet_manager.cc:1397] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.006s
I20250625 07:52:35.406910 1163 raft_consensus.cc:357] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.407111 1163 raft_consensus.cc:383] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.407197 1163 raft_consensus.cc:738] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.407496 1163 consensus_queue.cc:260] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.408030 1163 ts_tablet_manager.cc:1428] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:35.408198 2161 raft_consensus.cc:491] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.408342 1163 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.408335 2161 raft_consensus.cc:513] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.408921 2161 leader_election.cc:290] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.409662 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.409786 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.409859 908 raft_consensus.cc:2466] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:35.410225 713 leader_election.cc:343] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.410414 713 leader_election.cc:304] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.410660 2161 raft_consensus.cc:2802] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.410766 2161 raft_consensus.cc:491] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.410852 2161 raft_consensus.cc:3058] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.414117 2161 raft_consensus.cc:513] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.414790 2161 leader_election.cc:290] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.387717 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet da895c90d6ee40379fe8181c2dbd7a1e (table default.Table_444c92729289455eaf1a979360c97e5e [id=6fec84e2fefb4a738bce54d790e73177]) successfully deleted
I20250625 07:52:35.415292 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:35.415730 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "173a565916c1440493ece348599ddba3" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
W20250625 07:52:35.415820 713 leader_election.cc:343] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.415910 911 raft_consensus.cc:3058] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.417074 1163 tablet_bootstrap.cc:654] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.418107 2370 tablet.cc:1722] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:35.418779 911 raft_consensus.cc:2466] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:35.419087 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47d9a728e38f49cf948b3bdf5ff09621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.423200 713 leader_election.cc:304] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.423708 2161 raft_consensus.cc:2802] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.423862 2161 raft_consensus.cc:695] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.424127 2161 consensus_queue.cc:237] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.426009 1160 ts_tablet_manager.cc:1918] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.429651 1158 tablet_bootstrap.cc:492] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.429853 1158 ts_tablet_manager.cc:1397] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250625 07:52:35.430645 1158 raft_consensus.cc:357] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.430791 2112 raft_consensus.cc:491] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.430886 1158 raft_consensus.cc:383] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.430975 1158 raft_consensus.cc:738] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.430931 2112 raft_consensus.cc:513] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.431205 1158 consensus_queue.cc:260] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.431541 2112 leader_election.cc:290] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.431915 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.432157 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.432332 850 leader_election.cc:343] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.432868 851 leader_election.cc:343] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.432999 851 leader_election.cc:304] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.435113 2112 raft_consensus.cc:2747] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:35.438630 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6db07140365b4f2a8c6d6aaaf164be23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.443945 1163 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.444092 1163 ts_tablet_manager.cc:1397] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20250625 07:52:35.444890 1163 raft_consensus.cc:357] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.447914 1161 ts_tablet_manager.cc:1918] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.450371 1161 log.cc:1199] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/f9a94dc1432d4cbdbf3eecacb1429c23
I20250625 07:52:35.450760 1161 ts_tablet_manager.cc:1939] T f9a94dc1432d4cbdbf3eecacb1429c23 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.450906 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f442ff326d064398b181ffcb29aa6439" } new_table_name: "default.Table_81dacedf9d814b60986bc865cc7cbf94" modify_external_catalogs: true
I20250625 07:52:35.451496 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1178) took 1127 ms. Trace:
I20250625 07:52:35.445084 1163 raft_consensus.cc:383] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.447697 2112 raft_consensus.cc:491] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.451870 1163 raft_consensus.cc:738] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.451977 2112 raft_consensus.cc:513] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.426149 1160 log.cc:1199] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a41ffd566593499299aba5513ccb2f30
I20250625 07:52:35.452123 1163 consensus_queue.cc:260] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.452554 1160 ts_tablet_manager.cc:1939] T a41ffd566593499299aba5513ccb2f30 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:35.453382 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.451586 1161 rpcz_store.cc:276] 0625 07:52:34.323983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.324428 (+ 445us) service_pool.cc:225] Handling call
0625 07:52:35.358098 (+1033670us) ts_tablet_manager.cc:1145] Deleting tablet f9a94dc1432d4cbdbf3eecacb1429c23
0625 07:52:35.358108 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.371401 (+ 13293us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.447790 (+ 76389us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.451477 (+ 3687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027502}
W20250625 07:52:35.456379 851 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.456892 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.460592 1163 ts_tablet_manager.cc:1428] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:35.461247 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1179) took 1125 ms. Trace:
I20250625 07:52:35.461741 2112 leader_election.cc:290] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:35.461779 850 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.461890 850 leader_election.cc:304] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.462246 2112 raft_consensus.cc:2747] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.461573 1161 rpcz_store.cc:276] 0625 07:52:34.335981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.336496 (+ 515us) service_pool.cc:225] Handling call
0625 07:52:35.461045 (+1124549us) ts_tablet_manager.cc:1145] Deleting tablet 168f1a13533045d89ba1f07a72d4ff63
0625 07:52:35.461057 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.461207 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124344}
I20250625 07:52:35.464017 1161 tablet_replica.cc:331] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.465170 1161 raft_consensus.cc:2241] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.466140 1161 raft_consensus.cc:2270] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.469384 1163 tablet_bootstrap.cc:492] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.471524 2178 raft_consensus.cc:491] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.471663 2178 raft_consensus.cc:513] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.472234 2178 leader_election.cc:290] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.473410 986 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 1afe90251c964e369d05e82c7db19326
W20250625 07:52:35.473543 985 leader_election.cc:343] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1afe90251c964e369d05e82c7db19326
I20250625 07:52:35.473677 985 leader_election.cc:304] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.474037 2178 raft_consensus.cc:2747] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.456238 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet f9a94dc1432d4cbdbf3eecacb1429c23 (table default.Table_6a3fb30fbc5c482fbe1b7a3a1243fd95 [id=96ad1e3efde44205bc29c69277ed3eed]) successfully deleted
W20250625 07:52:35.474957 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 168f1a13533045d89ba1f07a72d4ff63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168f1a13533045d89ba1f07a72d4ff63 already in progress: creating tablet
I20250625 07:52:35.476401 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]). Expected version 1 got 0
I20250625 07:52:35.477725 1163 tablet_bootstrap.cc:654] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.478646 1161 ts_tablet_manager.cc:1905] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.480497 2161 raft_consensus.cc:491] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.480638 2161 raft_consensus.cc:513] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:35.481115 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4d308b0b16a4ab6a3a3589087c94136 consensus state has no leader
I20250625 07:52:35.481269 2161 leader_election.cc:290] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.481822 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b7daf759e324c1db853fdbaab76c721" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.482126 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4d062bccb2764b73a210a359c367f0cb" } new_table_name: "default.Table_9a780105a1f44d1f8ed934b8a1ee6b32" modify_external_catalogs: true
W20250625 07:52:35.482463 713 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0b7daf759e324c1db853fdbaab76c721
W20250625 07:52:35.483017 713 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.483137 713 leader_election.cc:304] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.483402 2161 raft_consensus.cc:2747] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.483456 1162 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.483618 1162 ts_tablet_manager.cc:1397] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.000s
I20250625 07:52:35.484344 1162 raft_consensus.cc:357] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.484650 1162 raft_consensus.cc:383] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.484755 1162 raft_consensus.cc:738] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.484982 1162 consensus_queue.cc:260] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.485514 1162 ts_tablet_manager.cc:1428] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:35.485639 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1209) took 1128 ms. Trace:
I20250625 07:52:35.485826 1162 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.485743 1160 rpcz_store.cc:276] 0625 07:52:34.357193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.357281 (+ 88us) service_pool.cc:225] Handling call
0625 07:52:35.387464 (+1030183us) ts_tablet_manager.cc:1145] Deleting tablet a41ffd566593499299aba5513ccb2f30
0625 07:52:35.387485 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.418846 (+ 31361us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.425903 (+ 7057us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.485576 (+ 59673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030005}
I20250625 07:52:35.486178 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a41ffd566593499299aba5513ccb2f30 (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:35.488391 1162 tablet_bootstrap.cc:654] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.491206 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1210) took 1116 ms. Trace:
I20250625 07:52:35.491307 1160 rpcz_store.cc:276] 0625 07:52:34.374315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.374368 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:35.491048 (+1116680us) ts_tablet_manager.cc:1145] Deleting tablet bff347d838fe4e56a195a957d0deb742
0625 07:52:35.491057 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.491150 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116480}
W20250625 07:52:35.491938 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bff347d838fe4e56a195a957d0deb742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff347d838fe4e56a195a957d0deb742 already in progress: creating tablet
I20250625 07:52:35.492574 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1211) took 1113 ms. Trace:
I20250625 07:52:35.492759 1160 rpcz_store.cc:276] 0625 07:52:34.378976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.379025 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:35.492424 (+1113399us) ts_tablet_manager.cc:1145] Deleting tablet bc387934e5204a3e9ce6f13d1dc3f777
0625 07:52:35.492434 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.492518 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113205}
W20250625 07:52:35.493207 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bc387934e5204a3e9ce6f13d1dc3f777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387934e5204a3e9ce6f13d1dc3f777 already in progress: creating tablet
I20250625 07:52:35.493546 1160 tablet_replica.cc:331] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:35.493796 1160 raft_consensus.cc:2241] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.494017 1160 raft_consensus.cc:2270] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.494578 1160 ts_tablet_manager.cc:1905] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.507758 1163 tablet_bootstrap.cc:492] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.508065 1163 ts_tablet_manager.cc:1397] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250625 07:52:35.508945 1163 raft_consensus.cc:357] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.509217 1163 raft_consensus.cc:383] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.509330 1163 raft_consensus.cc:738] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.509621 1163 consensus_queue.cc:260] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.510265 1163 ts_tablet_manager.cc:1428] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.510613 1163 tablet_bootstrap.cc:492] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.518939 2045 raft_consensus.cc:491] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.519207 2045 raft_consensus.cc:513] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.520982 2045 leader_election.cc:290] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.521615 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.522190 713 leader_election.cc:343] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.523819 1163 tablet_bootstrap.cc:654] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.524865 1161 ts_tablet_manager.cc:1918] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.524953 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.525251 911 raft_consensus.cc:2466] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.525761 713 leader_election.cc:304] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.526104 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_40d2beb409354e9ab21f0069193dbe48 [id=8c3e8751fb4c4d6b8e158a7851a5f2cb] alter complete (version 1)
I20250625 07:52:35.526206 2161 raft_consensus.cc:2802] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.526306 2161 raft_consensus.cc:491] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.526404 2161 raft_consensus.cc:3058] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.526782 1161 log.cc:1199] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/366d3e6fcc9d49be8fbd2cd070632861
I20250625 07:52:35.527074 1161 ts_tablet_manager.cc:1939] T 366d3e6fcc9d49be8fbd2cd070632861 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.528075 2112 raft_consensus.cc:491] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.528059 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f39da4a74c09429db8be207816d8f5d1" } new_table_name: "default.Table_1c20f484893f410e84bb744f4ac802a6" modify_external_catalogs: true
I20250625 07:52:35.528210 2112 raft_consensus.cc:513] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.529143 2161 raft_consensus.cc:513] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.529716 2161 leader_election.cc:290] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.531841 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5179a144a9a0445f933ca02e79bbc122" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.531945 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:35.532042 775 raft_consensus.cc:2466] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:35.532476 851 leader_election.cc:304] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:35.532765 2286 raft_consensus.cc:2802] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.532866 2286 raft_consensus.cc:491] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.532941 2286 raft_consensus.cc:3058] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.533396 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.533576 911 raft_consensus.cc:3058] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.535739 2286 raft_consensus.cc:513] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.537237 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5179a144a9a0445f933ca02e79bbc122" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:35.537797 850 leader_election.cc:343] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.538146 713 leader_election.cc:343] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.538443 2286 leader_election.cc:290] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.538479 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5179a144a9a0445f933ca02e79bbc122" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:35.538681 775 raft_consensus.cc:3058] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.538738 2112 leader_election.cc:290] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.539183 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5179a144a9a0445f933ca02e79bbc122" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.539585 850 leader_election.cc:343] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.541674 775 raft_consensus.cc:2466] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:35.542132 851 leader_election.cc:304] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.542513 2112 raft_consensus.cc:2802] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.542636 2112 raft_consensus.cc:695] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:35.542897 2112 consensus_queue.cc:237] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.553465 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1180) took 1217 ms. Trace:
I20250625 07:52:35.553599 1161 rpcz_store.cc:276] 0625 07:52:34.336349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.336599 (+ 250us) service_pool.cc:225] Handling call
0625 07:52:35.463661 (+1127062us) ts_tablet_manager.cc:1145] Deleting tablet 366d3e6fcc9d49be8fbd2cd070632861
0625 07:52:35.463827 (+ 166us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.503757 (+ 39930us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.524753 (+ 20996us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.553435 (+ 28682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126863}
I20250625 07:52:35.554172 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 366d3e6fcc9d49be8fbd2cd070632861 (table default.Table_3e1622741e834193aac4e19c1e66a6d4 [id=f87c26672d3b4fb1843098d895cb1a0e]) successfully deleted
I20250625 07:52:35.555214 593 catalog_manager.cc:5582] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
W20250625 07:52:35.555533 594 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:34495 (request call id 170) took 848 ms (client timeout 1000 ms). Trace:
W20250625 07:52:35.555632 594 rpcz_store.cc:269] 0625 07:52:34.706709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.706759 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:34.712355 (+ 5596us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 00007feeb473b14d 00007feeb473b243 00007feeb13fe341 00007feeb8f7fd8d 00007feeb91035f8 00007feeaff4cb36 00007feeaff4c865 00007feeac73ec7e 00007feeac73db5e 00007feeac743d95 00007feeac7446bf
0625 07:52:35.555508 (+843153us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":11428224}
I20250625 07:52:35.555747 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:35.556221 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_a9c6f935691440fe82ca615932a43240 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]). Expected version 1 got 0
I20250625 07:52:35.558215 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8d936848b9fc4ef384749a5a96ce3371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.558784 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d936848b9fc4ef384749a5a96ce3371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.559989 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.563091 1161 tablet_replica.cc:331] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.563280 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 08521c443c284f6aac4342b8ceb24d86
I20250625 07:52:35.563918 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 08521c443c284f6aac4342b8ceb24d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.564564 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 08521c443c284f6aac4342b8ceb24d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.566342 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 08521c443c284f6aac4342b8ceb24d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.563292 1161 raft_consensus.cc:2241] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.569446 1161 raft_consensus.cc:2270] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.570020 1161 ts_tablet_manager.cc:1905] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.571607 1160 ts_tablet_manager.cc:1918] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.571725 1160 log.cc:1199] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9cc1842f257d409ba0a56873f582f023
I20250625 07:52:35.572000 1160 ts_tablet_manager.cc:1939] T 9cc1842f257d409ba0a56873f582f023 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:35.572185 1162 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.572397 1162 ts_tablet_manager.cc:1397] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20250625 07:52:35.572844 2178 raft_consensus.cc:491] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.572970 2178 raft_consensus.cc:513] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.573576 2178 leader_election.cc:290] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.574365 985 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7e60d4f9aed44082b9905fbb44ffb110
W20250625 07:52:35.574751 986 leader_election.cc:343] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:35.574846 986 leader_election.cc:304] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.574851 1162 raft_consensus.cc:357] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:35.575019 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 08521c443c284f6aac4342b8ceb24d86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08521c443c284f6aac4342b8ceb24d86 already in progress: creating tablet
I20250625 07:52:35.575120 1162 raft_consensus.cc:383] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.575204 1162 raft_consensus.cc:738] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.575305 2178 raft_consensus.cc:2747] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.575512 1162 consensus_queue.cc:260] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.578928 1161 ts_tablet_manager.cc:1918] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.579074 1161 log.cc:1199] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/541c983d64f348539979f74199f79e59
I20250625 07:52:35.579411 1161 ts_tablet_manager.cc:1939] T 541c983d64f348539979f74199f79e59 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.580135 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1182) took 1243 ms. Trace:
I20250625 07:52:35.580266 1161 rpcz_store.cc:276] 0625 07:52:34.336898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.336954 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:35.562454 (+1225500us) ts_tablet_manager.cc:1145] Deleting tablet 541c983d64f348539979f74199f79e59
0625 07:52:35.563042 (+ 588us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.573329 (+ 10287us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.578832 (+ 5503us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.580118 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225096}
I20250625 07:52:35.580880 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 541c983d64f348539979f74199f79e59 (table default.Table_2cdfc224b12c489ebb46a9d55a90c823 [id=e92e25603ebb4d0d8000889777e06e56]) successfully deleted
I20250625 07:52:35.581125 2045 raft_consensus.cc:491] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.581271 2045 raft_consensus.cc:513] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.581490 1163 tablet_bootstrap.cc:492] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.581612 1163 ts_tablet_manager.cc:1397] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20250625 07:52:35.581915 2045 leader_election.cc:290] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.582274 1161 tablet_replica.cc:331] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.582381 1163 raft_consensus.cc:357] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.582547 1163 raft_consensus.cc:383] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.582607 1163 raft_consensus.cc:738] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.582511 1161 raft_consensus.cc:2241] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.582842 1161 raft_consensus.cc:2270] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.582806 1163 consensus_queue.cc:260] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.582868 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "716e0c4dab584788bbb7631b4ca934bf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.583376 1163 ts_tablet_manager.cc:1428] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.583482 1161 ts_tablet_manager.cc:1905] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.583645 1163 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.580753 1162 ts_tablet_manager.cc:1428] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250625 07:52:35.584189 713 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 716e0c4dab584788bbb7631b4ca934bf
W20250625 07:52:35.584414 713 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.584532 713 leader_election.cc:304] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.584823 2045 raft_consensus.cc:2747] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.585021 1163 tablet_bootstrap.cc:654] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.586658 1158 ts_tablet_manager.cc:1428] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.157s user 0.022s sys 0.000s
I20250625 07:52:35.587622 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ff6a95b9c737428fac9ea3ef1a5e8ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:35.587759 2340 tablet.cc:1722] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:35.588161 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6a95b9c737428fac9ea3ef1a5e8ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.589598 1158 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.590361 2345 raft_consensus.cc:491] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.590493 2345 raft_consensus.cc:513] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.590966 1158 tablet_bootstrap.cc:654] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.591056 2345 leader_election.cc:290] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.597652 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da895c90d6ee40379fe8181c2dbd7a1e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.598759 2112 raft_consensus.cc:491] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.598887 2112 raft_consensus.cc:513] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.599483 2112 leader_election.cc:290] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.599895 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70094e6c7e644f62bf5ef04dcfb4c380" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.600188 713 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: da895c90d6ee40379fe8181c2dbd7a1e
W20250625 07:52:35.600485 851 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.600564 713 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.600683 713 leader_election.cc:304] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.601073 2161 raft_consensus.cc:2747] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.601819 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70094e6c7e644f62bf5ef04dcfb4c380" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.602373 850 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.602464 850 leader_election.cc:304] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.603143 2112 raft_consensus.cc:2747] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.603492 1162 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.605525 2366 tablet.cc:1722] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:35.611060 2345 raft_consensus.cc:491] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.611256 2345 raft_consensus.cc:513] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.611968 2345 leader_election.cc:290] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.613000 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.613565 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.613735 908 raft_consensus.cc:2466] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.614514 713 leader_election.cc:304] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:35.616045 1162 tablet_bootstrap.cc:654] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.617046 1163 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.616096 2345 raft_consensus.cc:2802] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.617277 1163 ts_tablet_manager.cc:1397] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.003s
I20250625 07:52:35.617257 2345 raft_consensus.cc:491] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.617403 2345 raft_consensus.cc:3058] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.618191 1163 raft_consensus.cc:357] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.618438 1163 raft_consensus.cc:383] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.618530 1163 raft_consensus.cc:738] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.618772 1163 consensus_queue.cc:260] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:35.616256 713 leader_election.cc:343] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.619454 1163 ts_tablet_manager.cc:1428] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:35.619741 1163 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.620141 2345 raft_consensus.cc:513] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.621045 2345 leader_election.cc:290] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.621366 1163 tablet_bootstrap.cc:654] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.621801 2112 raft_consensus.cc:491] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.621687 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:35.621917 2112 raft_consensus.cc:513] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.622296 1161 ts_tablet_manager.cc:1918] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.622398 1161 log.cc:1199] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/5ec29e4294864fbabdff5a04dcf5b163
I20250625 07:52:35.622453 2112 leader_election.cc:290] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.622835 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a90bb52358f4c38a540ad025e9e0430" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.622915 1161 ts_tablet_manager.cc:1939] T 5ec29e4294864fbabdff5a04dcf5b163 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.621825 2286 raft_consensus.cc:491] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.623106 2286 raft_consensus.cc:513] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:35.623247 850 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.622721 713 leader_election.cc:343] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.623580 2286 leader_election.cc:290] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.623034 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a90bb52358f4c38a540ad025e9e0430" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.623824 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1181) took 1287 ms. Trace:
I20250625 07:52:35.623936 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d2bb8b0a8f45079083f2b42bc25011" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.624135 851 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.624225 851 leader_election.cc:304] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.623593 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fdc3220d53142faae41fd1f1500de56" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.624493 908 raft_consensus.cc:3058] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:35.624500 850 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.624732 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d2bb8b0a8f45079083f2b42bc25011" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.625110 2286 raft_consensus.cc:2747] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:35.625468 851 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.625547 851 leader_election.cc:304] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.625898 2286 raft_consensus.cc:2747] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.623889 1161 rpcz_store.cc:276] 0625 07:52:34.336786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.337527 (+ 741us) service_pool.cc:225] Handling call
0625 07:52:35.582227 (+1244700us) ts_tablet_manager.cc:1145] Deleting tablet 5ec29e4294864fbabdff5a04dcf5b163
0625 07:52:35.582238 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.612847 (+ 30609us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.622197 (+ 9350us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.623806 (+ 1609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244553}
I20250625 07:52:35.626536 1161 tablet_replica.cc:331] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.626662 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 5ec29e4294864fbabdff5a04dcf5b163 (table default.Table_3525675ad75a431abb04ceb227487582 [id=c17e0435b8b14760a4e8f7cf7887c9ff]) successfully deleted
I20250625 07:52:35.626827 1161 raft_consensus.cc:2241] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.627000 1161 raft_consensus.cc:2270] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:35.627653 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec2459d73e594a0c9622acc14bf1c5b5 consensus state has no leader
I20250625 07:52:35.627687 908 raft_consensus.cc:2466] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.627821 1161 ts_tablet_manager.cc:1905] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.628208 713 leader_election.cc:304] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.628516 2345 raft_consensus.cc:2802] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.628710 2345 raft_consensus.cc:695] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.628947 2345 consensus_queue.cc:237] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.634063 1158 tablet_bootstrap.cc:492] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.634477 1158 ts_tablet_manager.cc:1397] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.006s
I20250625 07:52:35.635619 1158 raft_consensus.cc:357] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.636371 1158 raft_consensus.cc:383] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.636523 1158 raft_consensus.cc:738] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.636925 1158 consensus_queue.cc:260] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.637746 2178 raft_consensus.cc:491] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.637879 2178 raft_consensus.cc:513] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.638453 2178 leader_election.cc:290] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.642215 1162 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.642385 1162 ts_tablet_manager.cc:1397] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250625 07:52:35.643010 1162 raft_consensus.cc:357] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.643209 2286 raft_consensus.cc:491] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.643316 2286 raft_consensus.cc:513] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.643838 2286 leader_election.cc:290] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.644246 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.644301 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.644678 1162 raft_consensus.cc:383] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:35.644748 850 leader_election.cc:343] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.644783 1162 raft_consensus.cc:738] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:35.644896 851 leader_election.cc:343] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.644981 851 leader_election.cc:304] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.645026 1162 consensus_queue.cc:260] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.645651 1162 ts_tablet_manager.cc:1428] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250625 07:52:35.645812 986 leader_election.cc:343] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: f191436f3b9244eaa12d211b6f125175
I20250625 07:52:35.645973 1162 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.646250 2286 raft_consensus.cc:2747] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.647608 1162 tablet_bootstrap.cc:654] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.648270 1163 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.648466 1163 ts_tablet_manager.cc:1397] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.003s
I20250625 07:52:35.649225 1163 raft_consensus.cc:357] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.649446 1163 raft_consensus.cc:383] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.649534 1163 raft_consensus.cc:738] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.649803 1163 consensus_queue.cc:260] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.650493 1163 ts_tablet_manager.cc:1428] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:35.650085 1158 ts_tablet_manager.cc:1428] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250625 07:52:35.654124 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0531838d7f4ceba54697f571889b52 (DEFAULT_TABLE table=default.Table_83efa7a2252740f7b29059fa6629975a [id=798a190ab5f14489a5a2a3303c615a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.654795 2161 raft_consensus.cc:491] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.654965 1158 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.654948 2161 raft_consensus.cc:513] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.655397 746 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0531838d7f4ceba54697f571889b52 (DEFAULT_TABLE table=default.Table_83efa7a2252740f7b29059fa6629975a [id=798a190ab5f14489a5a2a3303c615a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.655587 2161 leader_election.cc:290] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.655793 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0531838d7f4ceba54697f571889b52. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:35.658444 985 leader_election.cc:343] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f191436f3b9244eaa12d211b6f125175
I20250625 07:52:35.658576 985 leader_election.cc:304] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.658995 2178 raft_consensus.cc:2747] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.663044 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.663237 910 raft_consensus.cc:2466] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.663465 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.663559 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0531838d7f4ceba54697f571889b52. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:35.664214 713 leader_election.cc:343] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.664508 713 leader_election.cc:304] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.666194 2161 raft_consensus.cc:2802] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.666317 2161 raft_consensus.cc:491] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.666422 2161 raft_consensus.cc:3058] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.668182 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 10de48baf54c40449024eadbd264ba94
I20250625 07:52:35.669111 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 29a2f1c24e0240c0b1f0007961ee3ad2
I20250625 07:52:35.671226 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 10de48baf54c40449024eadbd264ba94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.671435 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.672008 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 10de48baf54c40449024eadbd264ba94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.672008 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:35.673066 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 81d13afd21f34e8794b603ccaa0b1d84 (table default.Table_5dc84a68d54248fe8ac0f3b4453374e0 [id=8ad1b460c1314a86a0d2de16d1f15f03], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 81d13afd21f34e8794b603ccaa0b1d84 consensus state has no leader
I20250625 07:52:35.674340 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:34 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.674575 2161 raft_consensus.cc:513] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:35.674909 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a2f1c24e0240c0b1f0007961ee3ad2 already in progress: creating tablet
I20250625 07:52:35.675431 2161 leader_election.cc:290] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.675707 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.675807 1161 ts_tablet_manager.cc:1918] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.675877 910 raft_consensus.cc:3058] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.675904 1161 log.cc:1199] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4c0bbd21d4aa41b2b92473bffe7dc05a
I20250625 07:52:35.676059 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:35.676187 1161 ts_tablet_manager.cc:1939] T 4c0bbd21d4aa41b2b92473bffe7dc05a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.676920 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1177) took 1353 ms. Trace:
I20250625 07:52:35.676990 1161 rpcz_store.cc:276] 0625 07:52:34.323630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.349257 (+ 25627us) service_pool.cc:225] Handling call
0625 07:52:35.626492 (+1277235us) ts_tablet_manager.cc:1145] Deleting tablet 4c0bbd21d4aa41b2b92473bffe7dc05a
0625 07:52:35.626502 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.631118 (+ 4616us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.675714 (+ 44596us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.676902 (+ 1188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277032}
W20250625 07:52:35.677091 713 leader_election.cc:343] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.677351 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1184) took 1297 ms. Trace:
I20250625 07:52:35.677418 1161 rpcz_store.cc:276] 0625 07:52:34.379626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.379685 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:35.677237 (+1297552us) ts_tablet_manager.cc:1145] Deleting tablet bc387934e5204a3e9ce6f13d1dc3f777
0625 07:52:35.677246 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.677321 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297342}
I20250625 07:52:35.677456 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 4c0bbd21d4aa41b2b92473bffe7dc05a (table default.Table_28a8c56dcb9b4bf093c733607ff00d69 [id=bfa6ee0232bc4576aeec2751dbe1a5c9]) successfully deleted
I20250625 07:52:35.677654 1161 tablet_replica.cc:331] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.677794 1161 raft_consensus.cc:2241] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:35.677803 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bc387934e5204a3e9ce6f13d1dc3f777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc387934e5204a3e9ce6f13d1dc3f777 already in progress: creating tablet
I20250625 07:52:35.677901 1161 raft_consensus.cc:2270] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.678314 1161 ts_tablet_manager.cc:1905] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.678848 910 raft_consensus.cc:2466] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.679303 713 leader_election.cc:304] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.679896 2045 raft_consensus.cc:2802] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.680054 2045 raft_consensus.cc:695] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.680282 2045 consensus_queue.cc:237] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.680739 2348 consensus_queue.cc:1035] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:35.683346 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 10de48baf54c40449024eadbd264ba94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.684495 1163 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:35.684495 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 10de48baf54c40449024eadbd264ba94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10de48baf54c40449024eadbd264ba94 already in progress: creating tablet
I20250625 07:52:35.686585 1163 tablet_bootstrap.cc:654] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.690549 1158 tablet_bootstrap.cc:654] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.687786 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1205) took 1375 ms. Trace:
I20250625 07:52:35.689260 879 tablet_service.cc:1468] Processing CreateTablet for tablet 7e0531838d7f4ceba54697f571889b52 (DEFAULT_TABLE table=default.Table_83efa7a2252740f7b29059fa6629975a [id=798a190ab5f14489a5a2a3303c615a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.691959 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e0531838d7f4ceba54697f571889b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.699115 1162 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.699326 1162 ts_tablet_manager.cc:1397] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.003s
I20250625 07:52:35.699239 1160 rpcz_store.cc:276] 0625 07:52:34.312392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.312457 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:35.493468 (+1181011us) ts_tablet_manager.cc:1145] Deleting tablet 9cc1842f257d409ba0a56873f582f023
0625 07:52:35.493478 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.497880 (+ 4402us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.571486 (+ 73606us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.687765 (+116279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088084}
I20250625 07:52:35.699679 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9cc1842f257d409ba0a56873f582f023 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:35.700084 1161 ts_tablet_manager.cc:1918] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.700202 1161 log.cc:1199] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/fc6c3be49be14ecbb3c26ef107ede015
I20250625 07:52:35.700503 1161 ts_tablet_manager.cc:1939] T fc6c3be49be14ecbb3c26ef107ede015 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.700664 1162 raft_consensus.cc:357] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.700881 1162 raft_consensus.cc:383] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.700976 1162 raft_consensus.cc:738] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.701196 1162 consensus_queue.cc:260] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.701715 1162 ts_tablet_manager.cc:1428] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:35.702167 1162 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.702582 1160 tablet_replica.cc:331] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:35.703898 1162 tablet_bootstrap.cc:654] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.706255 1160 raft_consensus.cc:2241] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.706401 2045 raft_consensus.cc:491] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.706554 2045 raft_consensus.cc:513] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.707242 2045 leader_election.cc:290] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.707207 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2e211834bf4a1591759658e0f8e2be (DEFAULT_TABLE table=default.Table_b975ec5e79ac42bfaa61d193c8ac170e [id=4b8fb08d2b1e4fc69d9e9add1a02befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.707619 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2e211834bf4a1591759658e0f8e2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.708112 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07a07af928ff4d3d894b074ef0cf44eb" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.709218 713 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.709530 713 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:35.709641 713 leader_election.cc:304] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.709890 2161 raft_consensus.cc:491] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.710031 2161 raft_consensus.cc:513] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.710611 2161 leader_election.cc:290] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.710908 2348 raft_consensus.cc:2747] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.711052 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7db12504a1a54add920561dae27d2d2b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.713167 713 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7db12504a1a54add920561dae27d2d2b
I20250625 07:52:35.707612 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1185) took 1318 ms. Trace:
I20250625 07:52:35.713749 1161 rpcz_store.cc:276] 0625 07:52:34.389339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.389555 (+ 216us) service_pool.cc:225] Handling call
0625 07:52:35.677610 (+1288055us) ts_tablet_manager.cc:1145] Deleting tablet fc6c3be49be14ecbb3c26ef107ede015
0625 07:52:35.677619 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.682519 (+ 4900us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.699992 (+ 17473us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.707591 (+ 7599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287716}
W20250625 07:52:35.713583 713 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.714265 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet fc6c3be49be14ecbb3c26ef107ede015 (table default.Table_ea09ec2fd2a34832ab3e312528a95048 [id=d68c36bb3c3145dc8699d0813c690388]) successfully deleted
I20250625 07:52:35.714304 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1183) took 1340 ms. Trace:
I20250625 07:52:35.714387 1161 rpcz_store.cc:276] 0625 07:52:34.373402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.413834 (+ 40432us) service_pool.cc:225] Handling call
0625 07:52:35.714201 (+1300367us) ts_tablet_manager.cc:1145] Deleting tablet bff347d838fe4e56a195a957d0deb742
0625 07:52:35.714210 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.714291 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300154}
I20250625 07:52:35.714645 1161 tablet_replica.cc:331] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.714816 1161 raft_consensus.cc:2241] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:35.714821 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bff347d838fe4e56a195a957d0deb742 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bff347d838fe4e56a195a957d0deb742 already in progress: creating tablet
I20250625 07:52:35.714952 1161 raft_consensus.cc:2270] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.715189 713 leader_election.cc:304] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.715474 2045 raft_consensus.cc:2747] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:35.715662 713 consensus_peers.cc:487] T 5199775108e44a9780009e4788a861ad P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:35.716161 746 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2e211834bf4a1591759658e0f8e2be (DEFAULT_TABLE table=default.Table_b975ec5e79ac42bfaa61d193c8ac170e [id=4b8fb08d2b1e4fc69d9e9add1a02befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.716647 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2e211834bf4a1591759658e0f8e2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.722931 1158 tablet_bootstrap.cc:492] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.723068 1158 ts_tablet_manager.cc:1397] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250625 07:52:35.711464 2286 raft_consensus.cc:491] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.723309 2286 raft_consensus.cc:513] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.723901 2286 leader_election.cc:290] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.723726 1158 raft_consensus.cc:357] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.721968 1163 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.724053 1158 raft_consensus.cc:383] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.724133 1158 raft_consensus.cc:738] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.724169 1163 ts_tablet_manager.cc:1397] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.006s
I20250625 07:52:35.724272 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.724341 1158 consensus_queue.cc:260] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.724452 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.724924 851 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.724973 1163 raft_consensus.cc:357] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.725123 1163 raft_consensus.cc:383] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.725172 1163 raft_consensus.cc:738] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.725201 1158 ts_tablet_manager.cc:1428] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250625 07:52:35.725343 1163 consensus_queue.cc:260] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:35.725728 850 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.725844 850 leader_election.cc:304] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.723310 1160 raft_consensus.cc:2270] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.726791 1162 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.727005 1162 ts_tablet_manager.cc:1397] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
I20250625 07:52:35.727238 1161 ts_tablet_manager.cc:1905] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.728497 1162 raft_consensus.cc:357] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.728946 1158 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.729096 1162 raft_consensus.cc:383] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.730726 1158 tablet_bootstrap.cc:654] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.730868 1162 raft_consensus.cc:738] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.731137 1162 consensus_queue.cc:260] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.735608 1162 ts_tablet_manager.cc:1428] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:35.737434 1162 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.739089 1162 tablet_bootstrap.cc:654] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.733835 1161 ts_tablet_manager.cc:1918] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.744776 1161 log.cc:1199] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/ba0735d576f947d395bee8d1b865f2fd
I20250625 07:52:35.744884 2286 raft_consensus.cc:2747] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.745123 1161 ts_tablet_manager.cc:1939] T ba0735d576f947d395bee8d1b865f2fd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.745426 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4b2e211834bf4a1591759658e0f8e2be (DEFAULT_TABLE table=default.Table_b975ec5e79ac42bfaa61d193c8ac170e [id=4b8fb08d2b1e4fc69d9e9add1a02befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.745656 1163 ts_tablet_manager.cc:1428] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250625 07:52:35.745817 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1186) took 1264 ms. Trace:
I20250625 07:52:35.745946 1161 rpcz_store.cc:276] 0625 07:52:34.481098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.485320 (+ 4222us) service_pool.cc:225] Handling call
0625 07:52:35.714606 (+1229286us) ts_tablet_manager.cc:1145] Deleting tablet ba0735d576f947d395bee8d1b865f2fd
0625 07:52:35.714613 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.730911 (+ 16298us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.733750 (+ 2839us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.745797 (+ 12047us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103045,"tablet-delete.queue_time_us":1229096}
I20250625 07:52:35.746397 2045 raft_consensus.cc:491] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.746567 2045 raft_consensus.cc:513] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.747385 2045 leader_election.cc:290] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.747946 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet ba0735d576f947d395bee8d1b865f2fd (table default.Table_97d416631de74a18be102771d2b2a6d5 [id=639abca5bb5a41f0b83e57ba15e39ac1]) successfully deleted
I20250625 07:52:35.748417 1161 tablet_replica.cc:331] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.748723 1161 raft_consensus.cc:2241] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.749032 1161 raft_consensus.cc:2270] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.749745 1161 ts_tablet_manager.cc:1905] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.750083 2178 raft_consensus.cc:491] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.750232 2178 raft_consensus.cc:513] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.750802 2178 leader_election.cc:290] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.751865 986 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:35.752142 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b2e211834bf4a1591759658e0f8e2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.752861 1163 tablet_bootstrap.cc:492] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:35.753238 985 leader_election.cc:343] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:35.753345 985 leader_election.cc:304] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:35.753579 2178 raft_consensus.cc:2747] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.754736 1163 tablet_bootstrap.cc:654] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.754714 1011 tablet_service.cc:1468] Processing CreateTablet for tablet dde275b2d99a4b0cb008a2fba9945960 (DEFAULT_TABLE table=default.Table_903b3fa5b06348ec8f2b92c77da28e7b [id=98304beb44c14c50ae404a85bb7cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.755155 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde275b2d99a4b0cb008a2fba9945960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.758433 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2172fec1936f46758baa42c0949361a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.758991 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2172fec1936f46758baa42c0949361a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.759814 594 master_service.cc:948] ReplaceTablet: received request to replace tablet fb2f623e8c9742c28a97ce3a4b16b002 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:35.760949 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_eaf3a4cc1ae44e4494084ce8431f9eaf" } new_table_name: "default.Table_90b0fb7fffff488ca122029f4343c687" modify_external_catalogs: true
I20250625 07:52:35.762009 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 7480c07af7ce4a3889bd8d42d7419d57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.762526 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8c8565ce30804a4f8148c11e83c7cce9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.762959 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c8565ce30804a4f8148c11e83c7cce9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.763006 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3d8b6b5c06248879e6052f6ae7fe46e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.763140 2112 raft_consensus.cc:491] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.763283 2112 raft_consensus.cc:513] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.763872 2112 leader_election.cc:290] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.764119 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3d8b6b5c06248879e6052f6ae7fe46e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.764708 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.765305 850 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.766265 758 tablet_service.cc:1468] Processing CreateTablet for tablet dde275b2d99a4b0cb008a2fba9945960 (DEFAULT_TABLE table=default.Table_903b3fa5b06348ec8f2b92c77da28e7b [id=98304beb44c14c50ae404a85bb7cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.766770 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde275b2d99a4b0cb008a2fba9945960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.766763 588 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:35.767088 588 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_a23788a3c209434db9f0ef44e0220672 [id=c80e78f1ef124af2b8f405e7d162ade8]). Expected version 2 got 0
I20250625 07:52:35.747041 1160 ts_tablet_manager.cc:1905] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.767221 588 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:35.767357 588 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_4bfac23414d34778bd98d79dcf630b42 [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 1 got 0
I20250625 07:52:35.771085 2112 raft_consensus.cc:491] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.771214 2112 raft_consensus.cc:513] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.771733 2112 leader_election.cc:290] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:35.775099 713 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.775462 713 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.775583 713 leader_election.cc:304] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.775844 2345 raft_consensus.cc:2747] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.776079 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d74924e8dee943e0ae4c93f0db1d75f4
I20250625 07:52:35.777236 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.777531 758 tablet_service.cc:1515] Processing DeleteTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.777877 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.778687 850 leader_election.cc:343] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:35.779052 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d74924e8dee943e0ae4c93f0db1d75f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d74924e8dee943e0ae4c93f0db1d75f4 already in progress: creating tablet
W20250625 07:52:35.781015 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4b87309b76f4f0e9dde120d9f755c44 consensus state has no leader
I20250625 07:52:35.783468 1161 ts_tablet_manager.cc:1918] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.783658 1161 log.cc:1199] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/2771159858fc41a1ad930229986edd92
I20250625 07:52:35.784636 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet cd728ffc2a274deea3fad533d1eaa6da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.784788 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f4359fd842a4760bae75169e8914f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.784581 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a737026476024d37af115083c4585882"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.785554 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a737026476024d37af115083c4585882 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.785955 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.786237 1161 ts_tablet_manager.cc:1939] T 2771159858fc41a1ad930229986edd92 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.786955 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.787318 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_aaf276d4a7fd405d964f4808679f5fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.787599 851 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.787981 851 leader_election.cc:304] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:35.788156 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaf276d4a7fd405d964f4808679f5fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.788362 2423 raft_consensus.cc:2747] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:35.789223 851 leader_election.cc:343] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.789798 1886 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "191b611b772a4fda9d1cba98776138a7" instance_seqno: 1750837930221676) as {username='slave'} at 127.0.104.1:44693; Asking this server to re-register.
I20250625 07:52:35.789840 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1187) took 1199 ms. Trace:
I20250625 07:52:35.789979 1162 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:35.789916 1161 rpcz_store.cc:276] 0625 07:52:34.589944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.589992 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:35.748353 (+1158361us) ts_tablet_manager.cc:1145] Deleting tablet 2771159858fc41a1ad930229986edd92
0625 07:52:35.748367 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.752932 (+ 4565us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.783369 (+ 30437us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.789821 (+ 6452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158158}
I20250625 07:52:35.790107 1162 ts_tablet_manager.cc:1397] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.004s
I20250625 07:52:35.790311 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 2771159858fc41a1ad930229986edd92 (table default.Table_4d1709d26e43407286b879f4c741db6e [id=783544279c5c41d2b9115617ea8b985c]) successfully deleted
I20250625 07:52:35.790666 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1188) took 1135 ms. Trace:
I20250625 07:52:35.790740 1161 rpcz_store.cc:276] 0625 07:52:34.655498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.655559 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:35.790552 (+1134993us) ts_tablet_manager.cc:1145] Deleting tablet 768c9551d33346718af2670a01a626b0
0625 07:52:35.790562 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.790651 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134795}
I20250625 07:52:35.790889 1162 raft_consensus.cc:357] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.791049 1162 raft_consensus.cc:383] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:35.791127 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 768c9551d33346718af2670a01a626b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 768c9551d33346718af2670a01a626b0 already in progress: creating tablet
I20250625 07:52:35.791661 1162 raft_consensus.cc:738] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.791910 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1189) took 1103 ms. Trace:
I20250625 07:52:35.792047 851 leader_election.cc:304] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.792008 1161 rpcz_store.cc:276] 0625 07:52:34.688912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.689370 (+ 458us) service_pool.cc:225] Handling call
0625 07:52:35.791792 (+1102422us) ts_tablet_manager.cc:1145] Deleting tablet d4d367cc123e4bbda8054d1d61d29be1
0625 07:52:35.791803 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.791891 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102253}
W20250625 07:52:35.792402 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d4d367cc123e4bbda8054d1d61d29be1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4d367cc123e4bbda8054d1d61d29be1 already in progress: creating tablet
I20250625 07:52:35.792469 2423 raft_consensus.cc:2747] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.792850 824 heartbeater.cc:461] Registering TS with master...
I20250625 07:52:35.793466 1158 tablet_bootstrap.cc:492] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.793572 1158 ts_tablet_manager.cc:1397] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.002s
I20250625 07:52:35.802390 2425 raft_consensus.cc:491] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3eb542ac24f6405687c1067f2885bec6)
I20250625 07:52:35.802717 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d74924e8dee943e0ae4c93f0db1d75f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:35 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.803014 2425 raft_consensus.cc:513] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } }
I20250625 07:52:35.804315 1162 consensus_queue.cc:260] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.804476 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1544 } } ignore_live_leader: false dest_uuid: "789c06e7de2842a1b40e97e0778ab6c9" is_pre_election: true
I20250625 07:52:35.805411 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00a3faeffaa4422ea73b98ee74768a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1544 } } ignore_live_leader: false dest_uuid: "3eb542ac24f6405687c1067f2885bec6" is_pre_election: true
I20250625 07:52:35.796576 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1190) took 1106 ms. Trace:
I20250625 07:52:35.806084 1161 rpcz_store.cc:276] 0625 07:52:34.690232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.690297 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:35.796474 (+1106177us) ts_tablet_manager.cc:1145] Deleting tablet b74d8f0eabf1410fadaadb98344b3f62
0625 07:52:35.796482 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.796560 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105978}
I20250625 07:52:35.806445 1885 ts_manager.cc:194] Registered new tserver with Master: 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:35.806533 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b74d8f0eabf1410fadaadb98344b3f62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74d8f0eabf1410fadaadb98344b3f62 already in progress: creating tablet
I20250625 07:52:35.806628 1820 leader_election.cc:304] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00a3faeffaa4422ea73b98ee74768a4a; no voters: 3eb542ac24f6405687c1067f2885bec6, 789c06e7de2842a1b40e97e0778ab6c9
I20250625 07:52:35.806931 1161 tablet_replica.cc:331] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.807235 1161 raft_consensus.cc:2241] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.807421 1158 raft_consensus.cc:357] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.807727 879 tablet_service.cc:1468] Processing CreateTablet for tablet dde275b2d99a4b0cb008a2fba9945960 (DEFAULT_TABLE table=default.Table_903b3fa5b06348ec8f2b92c77da28e7b [id=98304beb44c14c50ae404a85bb7cb67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.807951 1158 raft_consensus.cc:383] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.808058 1158 raft_consensus.cc:738] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.808143 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde275b2d99a4b0cb008a2fba9945960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.807657 1161 raft_consensus.cc:2270] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.808223 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 179e8bb43ab04f00a3317ee6dad257a2 (DEFAULT_TABLE table=default.Table_434924b9e2f44c0186b4b8958f24e778 [id=8761b0dc40c6481288d807ce0b9ec1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.808573 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e8bb43ab04f00a3317ee6dad257a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.808830 1161 ts_tablet_manager.cc:1905] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.810817 1163 tablet_bootstrap.cc:492] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.810997 1163 ts_tablet_manager.cc:1397] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20250625 07:52:35.811774 1163 raft_consensus.cc:357] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.811990 1163 raft_consensus.cc:383] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.812098 1163 raft_consensus.cc:738] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.812342 1163 consensus_queue.cc:260] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.812861 1163 ts_tablet_manager.cc:1428] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:35.814193 1158 consensus_queue.cc:260] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.816272 1160 ts_tablet_manager.cc:1918] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.820367 1158 ts_tablet_manager.cc:1428] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250625 07:52:35.825196 1158 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.826735 2425 leader_election.cc:290] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317), 3eb542ac24f6405687c1067f2885bec6 (127.0.104.60:33457)
I20250625 07:52:35.826793 1160 log.cc:1199] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c3d8b6b5c06248879e6052f6ae7fe46e
I20250625 07:52:35.819576 2345 raft_consensus.cc:491] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.827104 1160 ts_tablet_manager.cc:1939] T c3d8b6b5c06248879e6052f6ae7fe46e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:35.827069 2345 raft_consensus.cc:513] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.827394 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1212) took 1319 ms. Trace:
I20250625 07:52:35.827464 1160 rpcz_store.cc:276] 0625 07:52:34.507807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.507870 (+ 63us) service_pool.cc:225] Handling call
0625 07:52:35.702540 (+1194670us) ts_tablet_manager.cc:1145] Deleting tablet c3d8b6b5c06248879e6052f6ae7fe46e
0625 07:52:35.702549 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.779353 (+ 76804us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.816181 (+ 36828us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.827377 (+ 11196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20335,"tablet-delete.queue_time_us":1187705}
I20250625 07:52:35.827822 2345 leader_election.cc:290] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.827899 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c3d8b6b5c06248879e6052f6ae7fe46e (table default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]) successfully deleted
I20250625 07:52:35.822181 1163 tablet_bootstrap.cc:492] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.828011 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.828593 713 leader_election.cc:343] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.828770 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.828936 910 raft_consensus.cc:2466] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.829392 713 leader_election.cc:304] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.830014 1158 tablet_bootstrap.cc:654] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.830054 2425 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 00a3faeffaa4422ea73b98ee74768a4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 07:52:35.830571 2345 raft_consensus.cc:2802] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.830664 2345 raft_consensus.cc:491] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.830786 2345 raft_consensus.cc:3058] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.833392 1163 tablet_bootstrap.cc:654] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.833544 2345 raft_consensus.cc:513] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.834067 2423 raft_consensus.cc:491] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.834195 2423 raft_consensus.cc:513] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.834817 2423 leader_election.cc:290] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.835357 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96a1f823fd3b47c9b6d1cf5647f9a68a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.835552 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:35.835758 850 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.835940 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96a1f823fd3b47c9b6d1cf5647f9a68a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.835989 713 leader_election.cc:343] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.838495 1161 ts_tablet_manager.cc:1918] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.838644 2345 leader_election.cc:290] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:35.838989 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.839143 910 raft_consensus.cc:3058] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.840284 746 tablet_service.cc:1468] Processing CreateTablet for tablet 179e8bb43ab04f00a3317ee6dad257a2 (DEFAULT_TABLE table=default.Table_434924b9e2f44c0186b4b8958f24e778 [id=8761b0dc40c6481288d807ce0b9ec1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.840751 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e8bb43ab04f00a3317ee6dad257a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.841054 758 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb003c72bb54b7c84453083bdece85f (DEFAULT_TABLE table=default.Table_6af0d69ef30c4a8b92133ce2a3e3bb9f [id=e72561ba8f6644cd8c942e3416820189]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.841637 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb003c72bb54b7c84453083bdece85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.841758 910 raft_consensus.cc:2466] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.841907 754 tablet_service.cc:1468] Processing CreateTablet for tablet f07de62263b64352bd9423dc5f0a4e24 (DEFAULT_TABLE table=default.Table_53bbd615e4ad4b2a8def349b90e9cef8 [id=399dcc215aec49f784865d4568d9c747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.842190 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07de62263b64352bd9423dc5f0a4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.843219 1016 tablet_service.cc:1468] Processing CreateTablet for tablet f07de62263b64352bd9423dc5f0a4e24 (DEFAULT_TABLE table=default.Table_53bbd615e4ad4b2a8def349b90e9cef8 [id=399dcc215aec49f784865d4568d9c747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.843554 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07de62263b64352bd9423dc5f0a4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.844720 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d9b1ec100ca4488c9d7fc470246ffe47
I20250625 07:52:35.844898 752 tablet_service.cc:1468] Processing CreateTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f (DEFAULT_TABLE table=default.Table_f3b863e17f8942c58fe854c0dab420be [id=a01ea9fa448243398343ab7fe00ce550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.845109 590 catalog_manager.cc:6652] ReplaceTablet: tablet d9b1ec100ca4488c9d7fc470246ffe47 deleted and replaced by tablet 2f072505f55b4e59afde27106f2b3a36
I20250625 07:52:35.845191 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21be70ccbe434a1b89ae0dac8bc4347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.846473 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
I20250625 07:52:35.847141 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6157edbcfec44ec5a0e7ccd884de1ab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.847630 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6157edbcfec44ec5a0e7ccd884de1ab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.848228 713 leader_election.cc:304] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.848332 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb003c72bb54b7c84453083bdece85f (DEFAULT_TABLE table=default.Table_6af0d69ef30c4a8b92133ce2a3e3bb9f [id=e72561ba8f6644cd8c942e3416820189]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.848623 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_cd7eb16c71bb49cba917660f5a1c1b71 [id=bde8f673978c457e8a36b1173b315a06] alter complete (version 1)
I20250625 07:52:35.848711 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb003c72bb54b7c84453083bdece85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.838599 1161 log.cc:1199] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1898ff5b7626478bb997f7a433477498
I20250625 07:52:35.849112 1161 ts_tablet_manager.cc:1939] T 1898ff5b7626478bb997f7a433477498 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.849807 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1191) took 1127 ms. Trace:
I20250625 07:52:35.849881 1161 rpcz_store.cc:276] 0625 07:52:34.722557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.722777 (+ 220us) service_pool.cc:225] Handling call
0625 07:52:35.806874 (+1084097us) ts_tablet_manager.cc:1145] Deleting tablet 1898ff5b7626478bb997f7a433477498
0625 07:52:35.806886 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.811718 (+ 4832us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.838396 (+ 26678us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.849788 (+ 11392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083916}
I20250625 07:52:35.850996 893 tablet_service.cc:1468] Processing CreateTablet for tablet 179e8bb43ab04f00a3317ee6dad257a2 (DEFAULT_TABLE table=default.Table_434924b9e2f44c0186b4b8958f24e778 [id=8761b0dc40c6481288d807ce0b9ec1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.851303 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179e8bb43ab04f00a3317ee6dad257a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.851727 895 tablet_service.cc:1468] Processing CreateTablet for tablet f07de62263b64352bd9423dc5f0a4e24 (DEFAULT_TABLE table=default.Table_53bbd615e4ad4b2a8def349b90e9cef8 [id=399dcc215aec49f784865d4568d9c747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.852021 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f07de62263b64352bd9423dc5f0a4e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.854146 1161 tablet_replica.cc:331] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:35.854111 851 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.854218 851 leader_election.cc:304] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.854311 1161 raft_consensus.cc:2241] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.854418 2423 raft_consensus.cc:2747] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.854465 1161 raft_consensus.cc:2270] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.855852 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 1898ff5b7626478bb997f7a433477498 (table default.Table_8d5544c1432c4978ada968f7fcbfab59 [id=b0330ed5b54243c983112b2afaf66f1f]) successfully deleted
I20250625 07:52:35.856256 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f072505f55b4e59afde27106f2b3a36) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:35.856462 2423 raft_consensus.cc:491] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.856496 890 tablet_service.cc:1515] Processing DeleteTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f072505f55b4e59afde27106f2b3a36) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:35.856588 2423 raft_consensus.cc:513] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.843550 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f (DEFAULT_TABLE table=default.Table_f3b863e17f8942c58fe854c0dab420be [id=a01ea9fa448243398343ab7fe00ce550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.857170 2423 leader_election.cc:290] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.857389 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21be70ccbe434a1b89ae0dac8bc4347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.857769 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cdbdd1e3b734463ab452bafecb75822" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:35.858220 851 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.859689 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cdbdd1e3b734463ab452bafecb75822" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.861821 850 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.861920 850 leader_election.cc:304] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.862756 2423 raft_consensus.cc:2747] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.863224 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb003c72bb54b7c84453083bdece85f (DEFAULT_TABLE table=default.Table_6af0d69ef30c4a8b92133ce2a3e3bb9f [id=e72561ba8f6644cd8c942e3416820189]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.863576 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb003c72bb54b7c84453083bdece85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.863619 1161 ts_tablet_manager.cc:1905] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.864857 2161 raft_consensus.cc:2802] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.864979 2161 raft_consensus.cc:695] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.865192 2161 consensus_queue.cc:237] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.866097 2045 raft_consensus.cc:491] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.866204 2045 raft_consensus.cc:513] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.866724 2045 leader_election.cc:290] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.868328 1160 tablet_replica.cc:331] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:35.868546 1160 raft_consensus.cc:2241] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.868732 1160 raft_consensus.cc:2270] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.869313 1160 ts_tablet_manager.cc:1905] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.869326 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:35.870040 713 leader_election.cc:343] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.870486 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:35.870684 910 raft_consensus.cc:2466] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:35.870129 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d9b1ec100ca4488c9d7fc470246ffe47 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f072505f55b4e59afde27106f2b3a36) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:35.871892 713 leader_election.cc:304] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.872128 2045 raft_consensus.cc:2802] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:35.872200 2045 raft_consensus.cc:491] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:35.872252 2045 raft_consensus.cc:3058] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:35.872406 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d9b1ec100ca4488c9d7fc470246ffe47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9b1ec100ca4488c9d7fc470246ffe47 already in progress: creating tablet
I20250625 07:52:35.875751 2045 raft_consensus.cc:513] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.876747 2045 leader_election.cc:290] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.877024 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:35.877195 910 raft_consensus.cc:3058] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:35.877390 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:35.877856 713 leader_election.cc:343] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.879788 910 raft_consensus.cc:2466] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:35.881671 713 leader_election.cc:304] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.881919 2045 raft_consensus.cc:2802] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:35.882066 2045 raft_consensus.cc:695] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:35.882333 2045 consensus_queue.cc:237] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:35.883654 1161 ts_tablet_manager.cc:1918] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.883792 1161 log.cc:1199] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/cbdbd9230210455ba615d51f1a98267b
I20250625 07:52:35.884130 1161 ts_tablet_manager.cc:1939] T cbdbd9230210455ba615d51f1a98267b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.887584 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1192) took 1118 ms. Trace:
I20250625 07:52:35.887710 1161 rpcz_store.cc:276] 0625 07:52:34.768789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.768837 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:35.854099 (+1085262us) ts_tablet_manager.cc:1145] Deleting tablet cbdbd9230210455ba615d51f1a98267b
0625 07:52:35.854109 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.866576 (+ 12467us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.883519 (+ 16943us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.887560 (+ 4041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085086}
I20250625 07:52:35.888171 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet cbdbd9230210455ba615d51f1a98267b (table default.Table_dbf9b53cf6f843c1b02c9793e9bc0741 [id=acf404db716d454c970114267d40cd2f]) successfully deleted
W20250625 07:52:35.888993 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bdb6ad2e21814a5a829bea1a9bede857 has no consensus state
I20250625 07:52:35.889413 1161 tablet_replica.cc:331] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.889606 1161 raft_consensus.cc:2241] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.895434 2112 raft_consensus.cc:491] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.895572 2112 raft_consensus.cc:513] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.896122 2112 leader_election.cc:290] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:35.896763 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f47caf3bae41b79b0aa1e320f15da4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.896916 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f47caf3bae41b79b0aa1e320f15da4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:35.897202 1161 raft_consensus.cc:2270] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.897467 1163 tablet_bootstrap.cc:492] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:35.897508 851 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.897600 1163 ts_tablet_manager.cc:1397] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.000s
W20250625 07:52:35.897639 850 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.897706 850 leader_election.cc:304] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.897768 1161 ts_tablet_manager.cc:1905] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.897931 2112 raft_consensus.cc:2747] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.898361 1163 raft_consensus.cc:357] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.898535 1163 raft_consensus.cc:383] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.898588 1163 raft_consensus.cc:738] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.898792 1163 consensus_queue.cc:260] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.899367 1163 ts_tablet_manager.cc:1428] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:35.900254 1163 tablet_bootstrap.cc:492] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.901551 1162 ts_tablet_manager.cc:1428] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.111s user 0.000s sys 0.003s
I20250625 07:52:35.901726 1163 tablet_bootstrap.cc:654] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.901988 1162 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:35.904901 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e163a274311f4f969ff05cc5a7685b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.905454 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e163a274311f4f969ff05cc5a7685b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.907526 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
I20250625 07:52:35.908344 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_434924b9e2f44c0186b4b8958f24e778" } new_table_name: "default.Table_3c064735920d40be8f3021918d39181d" modify_external_catalogs: true
I20250625 07:52:35.911547 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 05d4e175016648e79c2ecb27ed578c84 (DEFAULT_TABLE table=default.Table_4aa9c46d0429418ab6f3300f000908c4 [id=74b77c47a6144b8f891b8e6108de2eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.911929 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d4e175016648e79c2ecb27ed578c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.910554 1161 ts_tablet_manager.cc:1918] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.917624 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a17495ae62b340258225c78df131b9c3 (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.917704 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 6fabcc19e6434c19b032677ed7204077 (DEFAULT_TABLE table=default.Table_ea49bfdf46d04bdf936e7c4cdf8af4d2 [id=65018a5b5ad549b49e6b713c6a3b4602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.917963 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17495ae62b340258225c78df131b9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.917704 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 09da1485a5b44aa1b42916f8a06a113f (DEFAULT_TABLE table=default.Table_596a78c3ff9e44158b80f0e0637d57ff [id=0bca508a0114413682998d763006b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.918371 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09da1485a5b44aa1b42916f8a06a113f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.919106 890 tablet_service.cc:1468] Processing CreateTablet for tablet 05d4e175016648e79c2ecb27ed578c84 (DEFAULT_TABLE table=default.Table_4aa9c46d0429418ab6f3300f000908c4 [id=74b77c47a6144b8f891b8e6108de2eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.919436 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d4e175016648e79c2ecb27ed578c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.920928 880 tablet_service.cc:1468] Processing CreateTablet for tablet 09da1485a5b44aa1b42916f8a06a113f (DEFAULT_TABLE table=default.Table_596a78c3ff9e44158b80f0e0637d57ff [id=0bca508a0114413682998d763006b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.921214 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09da1485a5b44aa1b42916f8a06a113f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.924839 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fabcc19e6434c19b032677ed7204077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.924962 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6fabcc19e6434c19b032677ed7204077 (DEFAULT_TABLE table=default.Table_ea49bfdf46d04bdf936e7c4cdf8af4d2 [id=65018a5b5ad549b49e6b713c6a3b4602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.925010 1161 log.cc:1199] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/be7b755d95b14ecea7a691003455f381
I20250625 07:52:35.925254 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fabcc19e6434c19b032677ed7204077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.925354 1161 ts_tablet_manager.cc:1939] T be7b755d95b14ecea7a691003455f381 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.926016 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1193) took 1104 ms. Trace:
I20250625 07:52:35.926090 1161 rpcz_store.cc:276] 0625 07:52:34.821522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.821764 (+ 242us) service_pool.cc:225] Handling call
0625 07:52:35.889350 (+1067586us) ts_tablet_manager.cc:1145] Deleting tablet be7b755d95b14ecea7a691003455f381
0625 07:52:35.889364 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.902589 (+ 13225us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.910450 (+ 7861us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.925999 (+ 15549us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1067427}
I20250625 07:52:35.926482 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet be7b755d95b14ecea7a691003455f381 (table default.Table_ca5ba7d48f0b44e9b52ef2de9e71ec0a [id=1095baa051544f968fc3065eddf77926]) successfully deleted
I20250625 07:52:35.928119 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1194) took 1055 ms. Trace:
I20250625 07:52:35.928190 1161 rpcz_store.cc:276] 0625 07:52:34.872446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.872498 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:35.928005 (+1055507us) ts_tablet_manager.cc:1145] Deleting tablet f326a912662f42e2abb74225853c112d
0625 07:52:35.928014 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.928103 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055313}
W20250625 07:52:35.928551 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f326a912662f42e2abb74225853c112d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326a912662f42e2abb74225853c112d already in progress: creating tablet
I20250625 07:52:35.928922 894 tablet_service.cc:1468] Processing CreateTablet for tablet a17495ae62b340258225c78df131b9c3 (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.929211 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17495ae62b340258225c78df131b9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.929316 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1195) took 1045 ms. Trace:
I20250625 07:52:35.929384 1161 rpcz_store.cc:276] 0625 07:52:34.883702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.883751 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:35.929212 (+1045461us) ts_tablet_manager.cc:1145] Deleting tablet f406d1c8777f474c97f6e804cb230bb6
0625 07:52:35.929220 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.929291 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045271}
W20250625 07:52:35.929703 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f406d1c8777f474c97f6e804cb230bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f406d1c8777f474c97f6e804cb230bb6 already in progress: creating tablet
I20250625 07:52:35.930325 1161 tablet_replica.cc:331] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.930490 1161 raft_consensus.cc:2241] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.930642 1161 raft_consensus.cc:2270] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.931882 1161 ts_tablet_manager.cc:1905] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.933007 895 tablet_service.cc:1468] Processing CreateTablet for tablet 53b34a2db6a345a6a22b28cd0435d95f (DEFAULT_TABLE table=default.Table_1463633d3ae24de887c29fc90b7c4c40 [id=4158e11be59b4f2c81343da8afbf10a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.933341 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b34a2db6a345a6a22b28cd0435d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.933465 1163 tablet_bootstrap.cc:492] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:35.933573 1163 ts_tablet_manager.cc:1397] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20250625 07:52:35.934182 1163 raft_consensus.cc:357] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.934334 1163 raft_consensus.cc:383] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.934379 1163 raft_consensus.cc:738] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.934659 893 tablet_service.cc:1468] Processing CreateTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f (DEFAULT_TABLE table=default.Table_f3b863e17f8942c58fe854c0dab420be [id=a01ea9fa448243398343ab7fe00ce550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.934957 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21be70ccbe434a1b89ae0dac8bc4347f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.934911 1163 consensus_queue.cc:260] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.935443 1163 ts_tablet_manager.cc:1428] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:35.935685 1163 tablet_bootstrap.cc:492] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:35.937381 1163 tablet_bootstrap.cc:654] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.939620 888 tablet_service.cc:1468] Processing CreateTablet for tablet 1463d047d40a49688267e9e396356841 (DEFAULT_TABLE table=default.Table_3aebb6670cd64a50af22d28be0a65e47 [id=246db39231624b71afe976911b03456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.939934 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1463d047d40a49688267e9e396356841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.942907 2045 raft_consensus.cc:491] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:35.943042 2045 raft_consensus.cc:513] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.943639 2045 leader_election.cc:290] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:35.944195 1820 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.60:33457, user_credentials={real_user=slave}} blocked reactor thread for 137878us
I20250625 07:52:35.944451 1158 tablet_bootstrap.cc:492] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.944818 1158 ts_tablet_manager.cc:1397] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.010s
I20250625 07:52:35.945488 1158 raft_consensus.cc:357] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.945639 1158 raft_consensus.cc:383] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.945681 1158 raft_consensus.cc:738] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.945848 1158 consensus_queue.cc:260] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.949488 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 1463d047d40a49688267e9e396356841 (DEFAULT_TABLE table=default.Table_3aebb6670cd64a50af22d28be0a65e47 [id=246db39231624b71afe976911b03456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.950531 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af8c114c1c464bcc95dbd287c88976d6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:35.951020 758 tablet_service.cc:1468] Processing CreateTablet for tablet 05d4e175016648e79c2ecb27ed578c84 (DEFAULT_TABLE table=default.Table_4aa9c46d0429418ab6f3300f000908c4 [id=74b77c47a6144b8f891b8e6108de2eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.951436 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d4e175016648e79c2ecb27ed578c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.951673 741 tablet_service.cc:1468] Processing CreateTablet for tablet 09da1485a5b44aa1b42916f8a06a113f (DEFAULT_TABLE table=default.Table_596a78c3ff9e44158b80f0e0637d57ff [id=0bca508a0114413682998d763006b9f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.952005 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09da1485a5b44aa1b42916f8a06a113f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.956959 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6fabcc19e6434c19b032677ed7204077 (DEFAULT_TABLE table=default.Table_ea49bfdf46d04bdf936e7c4cdf8af4d2 [id=65018a5b5ad549b49e6b713c6a3b4602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.957278 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fabcc19e6434c19b032677ed7204077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.960944 754 tablet_service.cc:1468] Processing CreateTablet for tablet 1463d047d40a49688267e9e396356841 (DEFAULT_TABLE table=default.Table_3aebb6670cd64a50af22d28be0a65e47 [id=246db39231624b71afe976911b03456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.961278 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1463d047d40a49688267e9e396356841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.961400 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1463d047d40a49688267e9e396356841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.961755 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9a0104b93e9a4a79b5140e24b91a74f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:35.962319 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a0104b93e9a4a79b5140e24b91a74f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:35.964924 749 tablet_service.cc:1468] Processing CreateTablet for tablet a17495ae62b340258225c78df131b9c3 (DEFAULT_TABLE table=default.Table_62b2ccb944794418a0e39d5874f5ef7e [id=689da0fa81174232bf0e83b311d363da]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:35.965241 713 leader_election.cc:343] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:35.965700 752 tablet_service.cc:1468] Processing CreateTablet for tablet 53b34a2db6a345a6a22b28cd0435d95f (DEFAULT_TABLE table=default.Table_1463633d3ae24de887c29fc90b7c4c40 [id=4158e11be59b4f2c81343da8afbf10a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.965258 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17495ae62b340258225c78df131b9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.966711 1158 ts_tablet_manager.cc:1428] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250625 07:52:35.968782 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b34a2db6a345a6a22b28cd0435d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.971519 1158 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:35.973121 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 53b34a2db6a345a6a22b28cd0435d95f (DEFAULT_TABLE table=default.Table_1463633d3ae24de887c29fc90b7c4c40 [id=4158e11be59b4f2c81343da8afbf10a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:35.973583 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b34a2db6a345a6a22b28cd0435d95f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:35.973698 1158 tablet_bootstrap.cc:654] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:35.979858 1160 ts_tablet_manager.cc:1918] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:35.980008 1160 log.cc:1199] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:35.980367 1160 ts_tablet_manager.cc:1939] T 8bc8f47541c54c4287ebb7b892daa1be P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:35.980675 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1213) took 1428 ms. Trace:
I20250625 07:52:35.980820 1160 rpcz_store.cc:276] 0625 07:52:34.552047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.552107 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:35.868277 (+1316170us) ts_tablet_manager.cc:1145] Deleting tablet 8bc8f47541c54c4287ebb7b892daa1be
0625 07:52:35.868287 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.886146 (+ 17859us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.979729 (+ 93583us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.980643 (+ 914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1315115}
I20250625 07:52:35.981042 1161 ts_tablet_manager.cc:1918] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250625 07:52:35.981101 713 leader_election.cc:343] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:35.981158 1161 log.cc:1199] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7fd497aa0b8c4838906524f4423c3756
I20250625 07:52:35.981217 713 leader_election.cc:304] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:35.981201 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8bc8f47541c54c4287ebb7b892daa1be (table default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]) successfully deleted
I20250625 07:52:35.981434 2345 consensus_queue.cc:1035] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:35.981545 1161 ts_tablet_manager.cc:1939] T 7fd497aa0b8c4838906524f4423c3756 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:35.981606 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1214) took 1325 ms. Trace:
I20250625 07:52:35.981709 1160 rpcz_store.cc:276] 0625 07:52:34.656059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.656112 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:35.981446 (+1325334us) ts_tablet_manager.cc:1145] Deleting tablet 768c9551d33346718af2670a01a626b0
0625 07:52:35.981455 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.981571 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325176}
I20250625 07:52:35.981841 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1196) took 1065 ms. Trace:
I20250625 07:52:35.981951 1161 rpcz_store.cc:276] 0625 07:52:34.916105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.916485 (+ 380us) service_pool.cc:225] Handling call
0625 07:52:35.930272 (+1013787us) ts_tablet_manager.cc:1145] Deleting tablet 7fd497aa0b8c4838906524f4423c3756
0625 07:52:35.930280 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.940231 (+ 9951us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.980923 (+ 40692us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:35.981809 (+ 886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013591}
W20250625 07:52:35.982072 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 768c9551d33346718af2670a01a626b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 768c9551d33346718af2670a01a626b0 already in progress: creating tablet
I20250625 07:52:35.982184 2345 raft_consensus.cc:2747] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:35.982276 1161 tablet_replica.cc:331] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:35.982275 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]) successfully deleted
I20250625 07:52:35.982491 1161 raft_consensus.cc:2241] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:35.982606 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1215) took 1293 ms. Trace:
I20250625 07:52:35.982650 1161 raft_consensus.cc:2270] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:35.982712 1160 rpcz_store.cc:276] 0625 07:52:34.689202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.689251 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:35.982480 (+1293229us) ts_tablet_manager.cc:1145] Deleting tablet d4d367cc123e4bbda8054d1d61d29be1
0625 07:52:35.982488 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.982564 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1293043}
I20250625 07:52:35.983028 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1216) took 1292 ms. Trace:
I20250625 07:52:35.983134 1160 rpcz_store.cc:276] 0625 07:52:34.690762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.690807 (+ 45us) service_pool.cc:225] Handling call
0625 07:52:35.982917 (+1292110us) ts_tablet_manager.cc:1145] Deleting tablet b74d8f0eabf1410fadaadb98344b3f62
0625 07:52:35.982924 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.982991 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291970}
I20250625 07:52:35.983223 1161 ts_tablet_manager.cc:1905] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:35.983461 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1217) took 1111 ms. Trace:
W20250625 07:52:35.983554 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d4d367cc123e4bbda8054d1d61d29be1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4d367cc123e4bbda8054d1d61d29be1 already in progress: creating tablet
I20250625 07:52:35.983577 1160 rpcz_store.cc:276] 0625 07:52:34.871936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.871995 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:35.983352 (+1111357us) ts_tablet_manager.cc:1145] Deleting tablet f326a912662f42e2abb74225853c112d
0625 07:52:35.983360 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.983427 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111180}
W20250625 07:52:35.983786 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b74d8f0eabf1410fadaadb98344b3f62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b74d8f0eabf1410fadaadb98344b3f62 already in progress: creating tablet
I20250625 07:52:35.983999 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1218) took 1101 ms. Trace:
W20250625 07:52:35.984007 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f326a912662f42e2abb74225853c112d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f326a912662f42e2abb74225853c112d already in progress: creating tablet
I20250625 07:52:35.984115 1160 rpcz_store.cc:276] 0625 07:52:34.882255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.882305 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:35.983890 (+1101585us) ts_tablet_manager.cc:1145] Deleting tablet f406d1c8777f474c97f6e804cb230bb6
0625 07:52:35.983898 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:35.983967 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101395}
W20250625 07:52:35.984570 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f406d1c8777f474c97f6e804cb230bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f406d1c8777f474c97f6e804cb230bb6 already in progress: creating tablet
I20250625 07:52:35.990717 1158 tablet_bootstrap.cc:492] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:35.990957 1158 ts_tablet_manager.cc:1397] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20250625 07:52:35.991744 1158 raft_consensus.cc:357] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.992937 1158 raft_consensus.cc:383] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:35.993099 1158 raft_consensus.cc:738] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:35.993435 1158 consensus_queue.cc:260] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:35.999872 1015 tablet_service.cc:1468] Processing CreateTablet for tablet b66d105d0395433a9862ca844f882919 (DEFAULT_TABLE table=default.Table_4db897967cc14088a2778777bce52c54 [id=ecc00f458525431d89d0fdd12b7e0e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.000330 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66d105d0395433a9862ca844f882919. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.002727 1158 ts_tablet_manager.cc:1428] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:36.005076 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d97f4c56eb574432af661966fcda939b (DEFAULT_TABLE table=default.Table_c9cb04ac9658429194e1997a689e4d2b [id=16d3a3518c25448baca5bfd2b63356d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.005529 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97f4c56eb574432af661966fcda939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.006268 2178 raft_consensus.cc:491] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.006450 2178 raft_consensus.cc:513] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.007009 2178 leader_election.cc:290] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.008554 985 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7480c07af7ce4a3889bd8d42d7419d57
W20250625 07:52:36.008962 986 leader_election.cc:343] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7480c07af7ce4a3889bd8d42d7419d57
I20250625 07:52:36.009083 986 leader_election.cc:304] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.009631 2178 raft_consensus.cc:2747] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.010401 752 tablet_service.cc:1468] Processing CreateTablet for tablet d97f4c56eb574432af661966fcda939b (DEFAULT_TABLE table=default.Table_c9cb04ac9658429194e1997a689e4d2b [id=16d3a3518c25448baca5bfd2b63356d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.010973 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97f4c56eb574432af661966fcda939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.012914 1161 ts_tablet_manager.cc:1918] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.013060 1161 log.cc:1199] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7e60d4f9aed44082b9905fbb44ffb110
I20250625 07:52:36.013378 1161 ts_tablet_manager.cc:1939] T 7e60d4f9aed44082b9905fbb44ffb110 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.013713 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1197) took 1079 ms. Trace:
I20250625 07:52:36.013803 1161 rpcz_store.cc:276] 0625 07:52:34.934488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.934540 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:35.982199 (+1047659us) ts_tablet_manager.cc:1145] Deleting tablet 7e60d4f9aed44082b9905fbb44ffb110
0625 07:52:35.982209 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.009757 (+ 27548us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.012815 (+ 3058us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.013666 (+ 851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047456}
I20250625 07:52:36.014230 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7e60d4f9aed44082b9905fbb44ffb110 (table default.Table_0f50983dec264e34b2ed511cd2925785 [id=6259f55094d745f8ba4c2814fa03fc60]) successfully deleted
I20250625 07:52:36.014623 1161 tablet_replica.cc:331] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.014858 1161 raft_consensus.cc:2241] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.015070 1161 raft_consensus.cc:2270] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.015609 1161 ts_tablet_manager.cc:1905] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.015646 894 tablet_service.cc:1468] Processing CreateTablet for tablet b66d105d0395433a9862ca844f882919 (DEFAULT_TABLE table=default.Table_4db897967cc14088a2778777bce52c54 [id=ecc00f458525431d89d0fdd12b7e0e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.017694 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5c8b736b762f4b4c996c442970f58a31 (DEFAULT_TABLE table=default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb [id=ac997b06b5b649c89e192ebf864d6e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.018096 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8b736b762f4b4c996c442970f58a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.018117 2045 raft_consensus.cc:491] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.018247 2045 raft_consensus.cc:513] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.018972 2045 leader_election.cc:290] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.019438 754 tablet_service.cc:1468] Processing CreateTablet for tablet cd226579c77b4938bb031fac0412cbb1 (DEFAULT_TABLE table=default.Table_e89d1c8b956c4d12a0b715252d470852 [id=a2b0c9228dc849c694b8c2478fd467f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.019805 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd226579c77b4938bb031fac0412cbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.022367 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.024019 1015 tablet_service.cc:1468] Processing CreateTablet for tablet cd226579c77b4938bb031fac0412cbb1 (DEFAULT_TABLE table=default.Table_e89d1c8b956c4d12a0b715252d470852 [id=a2b0c9228dc849c694b8c2478fd467f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.024453 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd226579c77b4938bb031fac0412cbb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:36.024464 713 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.024869 2161 raft_consensus.cc:491] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.024992 2161 raft_consensus.cc:513] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.025470 2161 leader_election.cc:290] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.026482 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.026894 713 leader_election.cc:343] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.027191 752 tablet_service.cc:1468] Processing CreateTablet for tablet b66d105d0395433a9862ca844f882919 (DEFAULT_TABLE table=default.Table_4db897967cc14088a2778777bce52c54 [id=ecc00f458525431d89d0fdd12b7e0e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.027551 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66d105d0395433a9862ca844f882919. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.032578 1162 tablet_bootstrap.cc:654] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.033644 1161 ts_tablet_manager.cc:1918] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.033773 1161 log.cc:1199] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/f191436f3b9244eaa12d211b6f125175
I20250625 07:52:36.038013 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fb2f623e8c9742c28a97ce3a4b16b002
I20250625 07:52:36.038513 594 catalog_manager.cc:6652] ReplaceTablet: tablet fb2f623e8c9742c28a97ce3a4b16b002 deleted and replaced by tablet 8551b83c436f47d1ad74d6a8e74486da
I20250625 07:52:36.039958 1161 ts_tablet_manager.cc:1939] T f191436f3b9244eaa12d211b6f125175 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.040336 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1198) took 1050 ms. Trace:
I20250625 07:52:36.040467 1161 rpcz_store.cc:276] 0625 07:52:34.990315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.990380 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:36.014539 (+1024159us) ts_tablet_manager.cc:1145] Deleting tablet f191436f3b9244eaa12d211b6f125175
0625 07:52:36.014548 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.030757 (+ 16209us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.033520 (+ 2763us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.040296 (+ 6776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023498}
I20250625 07:52:36.040886 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet f191436f3b9244eaa12d211b6f125175 (table default.Table_a8adac4095f54fafa0b25109890670ec [id=d3d5d9f995b04f1c82d9aee452c3f531]) successfully deleted
I20250625 07:52:36.041194 1161 tablet_replica.cc:331] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.041405 1161 raft_consensus.cc:2241] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.041558 1161 raft_consensus.cc:2270] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.042057 594 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:36.042281 1161 ts_tablet_manager.cc:1905] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.042977 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_893afa8db5b64a3fb5a0bcbea93d04aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.043509 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893afa8db5b64a3fb5a0bcbea93d04aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.047098 752 tablet_service.cc:1515] Processing DeleteTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 with delete_type TABLET_DATA_DELETED (replaced by tablet 8551b83c436f47d1ad74d6a8e74486da) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:36.047153 713 consensus_peers.cc:487] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:36.048811 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet fb2f623e8c9742c28a97ce3a4b16b002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2f623e8c9742c28a97ce3a4b16b002 already in progress: creating tablet
I20250625 07:52:36.054339 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.054598 914 raft_consensus.cc:2466] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:36.055379 1158 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.062461 2112 raft_consensus.cc:491] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.066298 1161 ts_tablet_manager.cc:1918] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.066524 1161 log.cc:1199] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3b9491eb269948a29888a7301ea75d42
I20250625 07:52:36.064972 713 leader_election.cc:304] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.070461 2045 raft_consensus.cc:2802] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:36.070595 2045 raft_consensus.cc:491] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:36.070691 2045 raft_consensus.cc:3058] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.074666 2045 raft_consensus.cc:513] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.075578 2045 leader_election.cc:290] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.076642 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:36.077776 713 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.089881 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.090179 909 raft_consensus.cc:2466] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:36.091028 713 leader_election.cc:304] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.091364 2045 raft_consensus.cc:2802] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:36.091478 2045 raft_consensus.cc:491] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:36.091558 2045 raft_consensus.cc:3058] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.092511 893 tablet_service.cc:1468] Processing CreateTablet for tablet d97f4c56eb574432af661966fcda939b (DEFAULT_TABLE table=default.Table_c9cb04ac9658429194e1997a689e4d2b [id=16d3a3518c25448baca5bfd2b63356d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.093070 888 tablet_service.cc:1468] Processing CreateTablet for tablet cd226579c77b4938bb031fac0412cbb1 (DEFAULT_TABLE table=default.Table_e89d1c8b956c4d12a0b715252d470852 [id=a2b0c9228dc849c694b8c2478fd467f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.094413 2045 raft_consensus.cc:513] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.095000 1160 tablet_replica.cc:331] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:36.095458 890 tablet_service.cc:1515] Processing DeleteTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 with delete_type TABLET_DATA_DELETED (replaced by tablet 8551b83c436f47d1ad74d6a8e74486da) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.095883 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b66d105d0395433a9862ca844f882919. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.095996 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:36.096673 2045 leader_election.cc:290] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.096909 713 leader_election.cc:343] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.097783 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bccda893fd5466886a4d253172309e1" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:36.098009 914 raft_consensus.cc:3058] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.102124 2112 raft_consensus.cc:513] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.102758 2112 leader_election.cc:290] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.103436 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.104462 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.105293 2423 raft_consensus.cc:491] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.105446 2423 raft_consensus.cc:513] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.105978 2423 leader_election.cc:290] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.106256 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:36.106416 909 raft_consensus.cc:3058] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.109222 909 raft_consensus.cc:2466] T c4729a62b7434cc0bcd1eecbfe838faf P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:36.109539 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97f4c56eb574432af661966fcda939b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.112622 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd226579c77b4938bb031fac0412cbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.112854 914 raft_consensus.cc:2466] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:36.113009 851 leader_election.cc:343] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.113806 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "034903d699584feb91f35cbb2f62701a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.114391 851 leader_election.cc:343] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.114820 850 leader_election.cc:343] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.115197 850 leader_election.cc:304] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.115845 713 leader_election.cc:304] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.116092 2045 raft_consensus.cc:2802] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:36.116257 713 leader_election.cc:304] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.116911 2161 raft_consensus.cc:2802] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:36.117026 2161 raft_consensus.cc:695] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:36.117228 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "034903d699584feb91f35cbb2f62701a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.117292 2161 consensus_queue.cc:237] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.117970 2423 raft_consensus.cc:2747] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.118296 2045 raft_consensus.cc:695] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
W20250625 07:52:36.118427 850 leader_election.cc:343] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.118535 850 leader_election.cc:304] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.118535 2045 consensus_queue.cc:237] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.119112 2423 raft_consensus.cc:2747] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:36.120656 713 consensus_peers.cc:487] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.125090 1160 raft_consensus.cc:2241] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.125456 1160 raft_consensus.cc:2270] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.126303 1160 ts_tablet_manager.cc:1905] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.128033 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5] alter complete (version 1)
I20250625 07:52:36.128641 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0ec61175463840a0a8f9a373e8f3d6fa" } modify_external_catalogs: true
I20250625 07:52:36.128818 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0ec61175463840a0a8f9a373e8f3d6fa" } modify_external_catalogs: true
W20250625 07:52:36.129014 593 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:33657 (request call id 226) took 1090 ms (client timeout 1000 ms). Trace:
W20250625 07:52:36.129094 593 rpcz_store.cc:269] 0625 07:52:35.038415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.038838 (+ 423us) service_pool.cc:225] Handling call
0625 07:52:36.128994 (+1090156us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:35.592905 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:35.592993 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:35.592997 (+ 4us) write_op.cc:435] Acquired schema lock
0625 07:52:35.592999 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:35.593069 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:35.593079 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:35.593084 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:35.593126 (+ 42us) tablet.cc:699] Row locks acquired
0625 07:52:35.593128 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:35.593152 (+ 24us) write_op.cc:270] Start()
0625 07:52:35.593174 (+ 22us) write_op.cc:276] Timestamp: P: 1750837955593136 usec, L: 0
0625 07:52:35.593178 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:35.593300 (+ 122us) log.cc:844] Serialized 527 byte log entry
0625 07:52:35.755375 (+162075us) op_driver.cc:464] REPLICATION: finished
0625 07:52:35.755532 (+ 157us) write_op.cc:301] APPLY: starting
0625 07:52:35.755546 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:35.755581 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:35.755609 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:35.755665 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:35.755699 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:35.755706 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:35.755782 (+ 76us) log.cc:844] Serialized 23 byte log entry
0625 07:52:35.755848 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:35.755865 (+ 17us) write_op.cc:454] Released schema lock
0625 07:52:35.755871 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:35.755950 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:35.755954 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0625 07:52:35.790958 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:35.791068 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:35.791074 (+ 6us) write_op.cc:435] Acquired schema lock
0625 07:52:35.791076 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:35.791156 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:35.791165 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:35.791169 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:35.791216 (+ 47us) tablet.cc:699] Row locks acquired
0625 07:52:35.791218 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:35.791247 (+ 29us) write_op.cc:270] Start()
0625 07:52:35.791273 (+ 26us) write_op.cc:276] Timestamp: P: 1750837955791229 usec, L: 0
0625 07:52:35.791277 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:35.791425 (+ 148us) log.cc:844] Serialized 356 byte log entry
0625 07:52:35.843020 (+ 51595us) op_driver.cc:464] REPLICATION: finished
0625 07:52:35.844209 (+ 1189us) write_op.cc:301] APPLY: starting
0625 07:52:35.844219 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:35.844246 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:35.844276 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:35.844311 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:35.844335 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:35.844341 (+ 6us) write_op.cc:312] APPLY: finished
0625 07:52:35.844394 (+ 53us) log.cc:844] Serialized 23 byte log entry
0625 07:52:35.844429 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:35.844442 (+ 13us) write_op.cc:454] Released schema lock
0625 07:52:35.844447 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:35.872846 (+ 28399us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:35.872853 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0625 07:52:35.966756 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:35.966835 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:35.966840 (+ 5us) write_op.cc:435] Acquired schema lock
0625 07:52:35.966842 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:35.966913 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:35.966920 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:35.966925 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:35.966965 (+ 40us) tablet.cc:699] Row locks acquired
0625 07:52:35.966967 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:35.966989 (+ 22us) write_op.cc:270] Start()
0625 07:52:35.967010 (+ 21us) write_op.cc:276] Timestamp: P: 1750837955966927 usec, L: 0
0625 07:52:35.967013 (+ 3us) op_driver.cc:348] REPLICATION: starting
0625 07:52:35.967124 (+ 111us) log.cc:844] Serialized 356 byte log entry
0625 07:52:36.125817 (+158693us) op_driver.cc:464] REPLICATION: finished
0625 07:52:36.126358 (+ 541us) write_op.cc:301] APPLY: starting
0625 07:52:36.126370 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:36.126425 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:36.126451 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:36.126497 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:36.126527 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:36.126534 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:36.126608 (+ 74us) log.cc:844] Serialized 23 byte log entry
0625 07:52:36.126653 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.126666 (+ 13us) write_op.cc:454] Released schema lock
0625 07:52:36.126673 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:36.126738 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.126742 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"num_ops":1,"prepare.queue_time_us":32644,"prepare.run_cpu_time_us":456,"prepare.run_wall_time_us":714,"replication_time_us":162166}],["op",{"apply.queue_time_us":1158,"apply.run_cpu_time_us":287,"apply.run_wall_time_us":288,"num_ops":1,"prepare.queue_time_us":33690,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":81875,"raft.queue_time_us":1053,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":676,"replication_time_us":51726}],["op",{"apply.queue_time_us":496,"apply.run_cpu_time_us":373,"apply.run_wall_time_us":373,"num_ops":1,"prepare.queue_time_us":116693,"prepare.run_cpu_time_us":430,"prepare.run_wall_time_us":767,"replication_time_us":158775}]]}
I20250625 07:52:36.130442 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6785410b52a4eb7a08b256bae5d35e0" } new_table_name: "default.Table_85a0ccf158584496bf48a2a6d82ded36" modify_external_catalogs: true
I20250625 07:52:36.131657 596 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2f072505f55b4e59afde27106f2b3a36
I20250625 07:52:36.131748 596 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34788 (request call id 1492) took 1059 ms. Trace:
I20250625 07:52:36.131816 596 rpcz_store.cc:276] 0625 07:52:35.072721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.073544 (+ 823us) service_pool.cc:225] Handling call
0625 07:52:35.074146 (+ 602us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 07:52:35.074291 (+ 145us) catalog_manager.cc:2790] Modifying in-memory table state
0625 07:52:35.074356 (+ 65us) catalog_manager.cc:2799] Locking tablets
0625 07:52:35.850257 (+775901us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 07:52:36.131517 (+281260us) catalog_manager.cc:2831] Removing table from by-name map
0625 07:52:36.131601 (+ 84us) catalog_manager.cc:2847] Committing in-memory state
0625 07:52:36.131632 (+ 31us) catalog_manager.cc:2851] Aborting table tasks
0625 07:52:36.131733 (+ 101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:35.967564 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:35.967668 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:35.967674 (+ 6us) write_op.cc:435] Acquired schema lock
0625 07:52:35.967677 (+ 3us) tablet.cc:660] Decoding operations
0625 07:52:35.967760 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:35.967767 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:35.967772 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0625 07:52:35.967848 (+ 76us) tablet.cc:699] Row locks acquired
0625 07:52:35.967850 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:35.967874 (+ 24us) write_op.cc:270] Start()
0625 07:52:35.967897 (+ 23us) write_op.cc:276] Timestamp: P: 1750837955967811 usec, L: 0
0625 07:52:35.967900 (+ 3us) op_driver.cc:348] REPLICATION: starting
0625 07:52:35.968018 (+ 118us) log.cc:844] Serialized 725 byte log entry
0625 07:52:36.125888 (+157870us) op_driver.cc:464] REPLICATION: finished
0625 07:52:36.126801 (+ 913us) write_op.cc:301] APPLY: starting
0625 07:52:36.126811 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:36.126844 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:36.126868 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:36.126930 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:36.126954 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 07:52:36.126963 (+ 9us) write_op.cc:312] APPLY: finished
0625 07:52:36.127010 (+ 47us) log.cc:844] Serialized 29 byte log entry
0625 07:52:36.127048 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.127061 (+ 13us) write_op.cc:454] Released schema lock
0625 07:52:36.127066 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:36.127132 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.127136 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":881,"apply.run_cpu_time_us":323,"apply.run_wall_time_us":322,"num_ops":2,"prepare.queue_time_us":116787,"prepare.run_cpu_time_us":517,"prepare.run_wall_time_us":767,"replication_time_us":157970}]]}
I20250625 07:52:36.132541 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5c1c689b7e634891a7e82c704c095684" } modify_external_catalogs: true
I20250625 07:52:36.132714 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5c1c689b7e634891a7e82c704c095684" } modify_external_catalogs: true
I20250625 07:52:36.133672 2178 raft_consensus.cc:491] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.133837 2178 raft_consensus.cc:513] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:36.133927 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 179e8bb43ab04f00a3317ee6dad257a2 consensus state has no leader
I20250625 07:52:36.134253 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_94e2dfea1b4a4e2aaf49eefe8dd523f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:36.134438 2178 leader_election.cc:290] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.134689 1160 ts_tablet_manager.cc:1918] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250625 07:52:36.134737 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94e2dfea1b4a4e2aaf49eefe8dd523f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:36.135524 985 leader_election.cc:343] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9f4359fd842a4760bae75169e8914f1c
W20250625 07:52:36.135813 986 leader_election.cc:343] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9f4359fd842a4760bae75169e8914f1c
I20250625 07:52:36.135926 986 leader_election.cc:304] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.136085 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e54e9c8d68174135872e7b1d2d1086a4" } modify_external_catalogs: true
I20250625 07:52:36.136214 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e54e9c8d68174135872e7b1d2d1086a4" } modify_external_catalogs: true
I20250625 07:52:36.136421 2178 raft_consensus.cc:2747] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.137655 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.137863 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.138022 2407 tablet_replica.cc:331] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:36.138188 2407 raft_consensus.cc:2241] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.138326 2407 raft_consensus.cc:2270] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.138797 2407 ts_tablet_manager.cc:1905] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.138985 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bb75ba8ad3674db084f085ca2c573442" } new_table_name: "default.Table_5d758b9980b7479a9ed999415a195e30" modify_external_catalogs: true
I20250625 07:52:36.140794 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.140988 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.141022 758 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.141131 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.141789 595 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:36.141948 595 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]). Expected version 1 got 0
I20250625 07:52:36.142932 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5e0793ddef83409a952052783347dad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.143414 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e0793ddef83409a952052783347dad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.143956 1160 log.cc:1199] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7fd497aa0b8c4838906524f4423c3756
I20250625 07:52:36.144374 1160 ts_tablet_manager.cc:1939] T 7fd497aa0b8c4838906524f4423c3756 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:36.146116 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1219) took 1235 ms. Trace:
I20250625 07:52:36.146241 1160 rpcz_store.cc:276] 0625 07:52:34.909103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:34.909156 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:36.055042 (+1145886us) ts_tablet_manager.cc:1145] Deleting tablet 7fd497aa0b8c4838906524f4423c3756
0625 07:52:36.055053 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.129978 (+ 74925us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.134572 (+ 4594us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.144745 (+ 10173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4838,"tablet-delete.queue_time_us":1145705}
I20250625 07:52:36.147439 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7fd497aa0b8c4838906524f4423c3756 (table default.Table_c96df891fb1343df8cc90ee2304287fb [id=368053320f3844fbbd5b935b9842499a]) successfully deleted
I20250625 07:52:36.147904 1160 tablet_replica.cc:331] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:36.148092 1160 raft_consensus.cc:2241] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.148275 1160 raft_consensus.cc:2270] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.148789 2407 ts_tablet_manager.cc:1918] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.148895 2407 log.cc:1199] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:36.148880 1160 ts_tablet_manager.cc:1905] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.149173 2407 ts_tablet_manager.cc:1939] T 0657163b9a0d4f7f99f57f7fd1f32462 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:36.150678 2407 tablet_replica.cc:331] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:36.150851 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0657163b9a0d4f7f99f57f7fd1f32462 (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
I20250625 07:52:36.159420 752 tablet_service.cc:1468] Processing CreateTablet for tablet 5c8b736b762f4b4c996c442970f58a31 (DEFAULT_TABLE table=default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb [id=ac997b06b5b649c89e192ebf864d6e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.159816 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8b736b762f4b4c996c442970f58a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.159794 1049 raft_consensus.cc:3058] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.169090 2407 raft_consensus.cc:2241] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:36.169303 1016 tablet_service.cc:1468] Processing CreateTablet for tablet b6c3420e04e2408f814fa832e8904d51 (DEFAULT_TABLE table=default.Table_d9b9ca707e724747aa3b189289a21de4 [id=0f57ae56973e4107b9c94567d6a104b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.169778 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c3420e04e2408f814fa832e8904d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.170405 894 tablet_service.cc:1468] Processing CreateTablet for tablet 5c8b736b762f4b4c996c442970f58a31 (DEFAULT_TABLE table=default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb [id=ac997b06b5b649c89e192ebf864d6e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.170732 2407 pending_rounds.cc:70] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:36.170773 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c8b736b762f4b4c996c442970f58a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.170795 2407 pending_rounds.cc:77] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432265310023680 op_type: NO_OP noop_request { }
I20250625 07:52:36.170886 2407 raft_consensus.cc:2887] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:36.170962 2407 raft_consensus.cc:2270] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.169636 752 tablet_service.cc:1468] Processing CreateTablet for tablet b6c3420e04e2408f814fa832e8904d51 (DEFAULT_TABLE table=default.Table_d9b9ca707e724747aa3b189289a21de4 [id=0f57ae56973e4107b9c94567d6a104b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.171460 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c3420e04e2408f814fa832e8904d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.171994 2407 ts_tablet_manager.cc:1905] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.178074 1163 tablet_bootstrap.cc:492] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.180938 1163 ts_tablet_manager.cc:1397] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.245s user 0.020s sys 0.039s
I20250625 07:52:36.181835 1163 raft_consensus.cc:357] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.187287 1163 raft_consensus.cc:383] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.187634 1163 raft_consensus.cc:738] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.188318 1163 consensus_queue.cc:260] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.189563 1163 ts_tablet_manager.cc:1428] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250625 07:52:36.190186 1163 tablet_bootstrap.cc:492] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.192057 1163 tablet_bootstrap.cc:654] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.193832 2045 consensus_queue.cc:1035] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.203047 1162 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.203307 1162 ts_tablet_manager.cc:1397] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.301s user 0.004s sys 0.005s
I20250625 07:52:36.204224 1162 raft_consensus.cc:357] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.206274 1162 raft_consensus.cc:383] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.206609 1162 raft_consensus.cc:738] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.207010 1162 consensus_queue.cc:260] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.207613 1162 ts_tablet_manager.cc:1428] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:36.213164 1162 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.214094 752 tablet_service.cc:1468] Processing CreateTablet for tablet 37d228244cca4052889acdc8f1ddd440 (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.214553 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d228244cca4052889acdc8f1ddd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.214803 893 tablet_service.cc:1468] Processing CreateTablet for tablet 37d228244cca4052889acdc8f1ddd440 (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.215148 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d228244cca4052889acdc8f1ddd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.215956 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 37d228244cca4052889acdc8f1ddd440 (DEFAULT_TABLE table=default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.216372 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37d228244cca4052889acdc8f1ddd440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.220893 1158 tablet_bootstrap.cc:654] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.220933 1161 ts_tablet_manager.cc:1939] T 3b9491eb269948a29888a7301ea75d42 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.220973 758 tablet_service.cc:1468] Processing CreateTablet for tablet 2e19c898ef7943bc8a40a9088a44bb2b (DEFAULT_TABLE table=default.Table_1d2d236daca14dee8585a320d9045d88 [id=386fc55a89ff4ef789f6d20020c98166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.221019 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 2e19c898ef7943bc8a40a9088a44bb2b (DEFAULT_TABLE table=default.Table_1d2d236daca14dee8585a320d9045d88 [id=386fc55a89ff4ef789f6d20020c98166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.221295 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e19c898ef7943bc8a40a9088a44bb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.221374 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1199) took 1181 ms. Trace:
I20250625 07:52:36.221479 1161 rpcz_store.cc:276] 0625 07:52:35.039431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.039478 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:36.041137 (+1001659us) ts_tablet_manager.cc:1145] Deleting tablet 3b9491eb269948a29888a7301ea75d42
0625 07:52:36.041146 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.052960 (+ 11814us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.056549 (+ 3589us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.221300 (+164751us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001477}
I20250625 07:52:36.221678 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2e19c898ef7943bc8a40a9088a44bb2b (DEFAULT_TABLE table=default.Table_1d2d236daca14dee8585a320d9045d88 [id=386fc55a89ff4ef789f6d20020c98166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.221864 1161 tablet_replica.cc:331] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.222012 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e19c898ef7943bc8a40a9088a44bb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.222064 1161 raft_consensus.cc:2241] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.222244 1161 raft_consensus.cc:2270] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.222810 1161 ts_tablet_manager.cc:1905] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.221376 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e19c898ef7943bc8a40a9088a44bb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.223855 1162 tablet_bootstrap.cc:654] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.224927 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3b9491eb269948a29888a7301ea75d42 (table default.Table_fc8939aafc524fba85e19aefef0e51bc [id=5431a24f6adf407b9ea8daaa4ac83e88]) successfully deleted
I20250625 07:52:36.180680 1160 ts_tablet_manager.cc:1918] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.236541 1160 log.cc:1199] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:36.236917 1160 ts_tablet_manager.cc:1939] T 9f1d251171314118b66e3a921a4a50c5 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:36.237529 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9f1d251171314118b66e3a921a4a50c5 (table default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]) successfully deleted
W20250625 07:52:36.240267 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 08521c443c284f6aac4342b8ceb24d86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08521c443c284f6aac4342b8ceb24d86 already in progress: creating tablet
I20250625 07:52:36.240763 1162 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.241003 1162 ts_tablet_manager.cc:1397] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
W20250625 07:52:36.240960 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 10de48baf54c40449024eadbd264ba94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10de48baf54c40449024eadbd264ba94 already in progress: creating tablet
W20250625 07:52:36.242028 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a2f1c24e0240c0b1f0007961ee3ad2 already in progress: creating tablet
I20250625 07:52:36.242056 1162 raft_consensus.cc:357] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.242235 1162 raft_consensus.cc:383] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.242324 1162 raft_consensus.cc:738] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:36.242626 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d74924e8dee943e0ae4c93f0db1d75f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d74924e8dee943e0ae4c93f0db1d75f4 already in progress: creating tablet
I20250625 07:52:36.242580 1162 consensus_queue.cc:260] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:36.242931 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fb2f623e8c9742c28a97ce3a4b16b002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2f623e8c9742c28a97ce3a4b16b002 already in progress: creating tablet
I20250625 07:52:36.243371 1161 ts_tablet_manager.cc:1918] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.243582 1161 log.cc:1199] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/743123ffcf8240dc82bea7178559b7ea
I20250625 07:52:36.243777 2443 tablet.cc:1722] T c5fe3ea5745743019d23d49ff808070a P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:36.247443 1162 ts_tablet_manager.cc:1428] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:36.247745 1162 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:36.250798 911 consensus_meta.cc:287] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Time spent flushing consensus metadata: real 0.717s user 0.024s sys 0.001s
I20250625 07:52:36.251479 1162 tablet_bootstrap.cc:654] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:36.266561 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95545f3569ad49828ae49136f122e690 consensus state has no leader
I20250625 07:52:36.269660 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b297f8cfc30a4f8cabed1ece38c77f00" } modify_external_catalogs: true
I20250625 07:52:36.269857 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b297f8cfc30a4f8cabed1ece38c77f00" } modify_external_catalogs: true
I20250625 07:52:36.269867 591 catalog_manager.cc:5582] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.274361 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_a9c6f935691440fe82ca615932a43240 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]). Expected version 1 got 0
I20250625 07:52:36.274729 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:36.275177 591 catalog_manager.cc:5582] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.275892 591 catalog_manager.cc:5582] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.276280 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:36.276775 591 catalog_manager.cc:5582] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.277155 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91]). Expected version 2 got 0
I20250625 07:52:36.277511 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:36.279464 591 catalog_manager.cc:5582] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.279882 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 4da889781dbd41aebac86e9833cf2b7e (table default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba]). Expected version 1 got 0
I20250625 07:52:36.280427 591 catalog_manager.cc:5582] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.281018 591 catalog_manager.cc:5582] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:36.281404 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:36.281808 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 28e490ba5d08498f921b50f057b88369
I20250625 07:52:36.282472 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bcbc8715a04246c4a1760354c9e1f45d" } new_table_name: "default.Table_e7eadb1ed2af45bf8ebc908ee6e8b073" modify_external_catalogs: true
I20250625 07:52:36.282927 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 28e490ba5d08498f921b50f057b88369 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.244009 1161 ts_tablet_manager.cc:1939] T 743123ffcf8240dc82bea7178559b7ea P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.255587 2178 raft_consensus.cc:491] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.298056 2178 raft_consensus.cc:513] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.298877 2178 leader_election.cc:290] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.300031 986 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: cd728ffc2a274deea3fad533d1eaa6da
W20250625 07:52:36.300990 985 leader_election.cc:343] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:36.301132 985 leader_election.cc:304] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.301681 2178 raft_consensus.cc:2747] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.302232 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 743123ffcf8240dc82bea7178559b7ea (table default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]) successfully deleted
I20250625 07:52:36.284865 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 28e490ba5d08498f921b50f057b88369 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.302853 1161 tablet_replica.cc:331] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.286327 2447 raft_consensus.cc:491] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.303570 2447 raft_consensus.cc:513] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.304807 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.303798 1161 raft_consensus.cc:2241] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.305565 1161 raft_consensus.cc:2270] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.306178 1161 ts_tablet_manager.cc:1905] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.306411 2161 raft_consensus.cc:491] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.306593 2161 raft_consensus.cc:513] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.307360 2161 leader_election.cc:290] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.308372 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f1d251171314118b66e3a921a4a50c5" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.308676 1048 raft_consensus.cc:3058] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.308612 2447 leader_election.cc:290] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.313076 713 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.313300 713 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:36.313410 713 leader_election.cc:304] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.313632 2045 raft_consensus.cc:2747] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.320837 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4] alter complete (version 3)
I20250625 07:52:36.330825 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f1a22d57d846414a8fa5855dc542b594
I20250625 07:52:36.331523 596 master_service.cc:948] ReplaceTablet: received request to replace tablet c82b840442ec4cb4993636183812ed22 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:36.335924 758 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a22d57d846414a8fa5855dc542b594 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.336582 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a22d57d846414a8fa5855dc542b594 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.348217 1158 tablet_bootstrap.cc:492] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:36.348547 1158 ts_tablet_manager.cc:1397] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.342s user 0.004s sys 0.000s
I20250625 07:52:36.365969 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_73c6562565d24a9abf20fbef500de416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.366560 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73c6562565d24a9abf20fbef500de416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:36.372504 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09120e19ea0f4ec7b94c89d173d3609f consensus state has no leader
I20250625 07:52:36.373519 1158 raft_consensus.cc:357] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.373778 1158 raft_consensus.cc:383] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.373850 1158 raft_consensus.cc:738] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.374159 1158 consensus_queue.cc:260] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.382628 1158 ts_tablet_manager.cc:1428] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250625 07:52:36.384459 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7849c32289204dd5a9024a89fd91292c" } new_table_name: "default.Table_3aaa21f170574b79ae5d89ff2cc423ad" modify_external_catalogs: true
I20250625 07:52:36.387799 1158 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.391906 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet fb2f623e8c9742c28a97ce3a4b16b002 with delete_type TABLET_DATA_DELETED (replaced by tablet 8551b83c436f47d1ad74d6a8e74486da) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.393728 2407 ts_tablet_manager.cc:1918] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:36.398010 2407 log.cc:1199] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9cc1842f257d409ba0a56873f582f023
I20250625 07:52:36.398672 2407 ts_tablet_manager.cc:1939] T 9cc1842f257d409ba0a56873f582f023 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:36.399933 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9cc1842f257d409ba0a56873f582f023 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:36.400318 2407 tablet_replica.cc:331] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:36.400576 2407 raft_consensus.cc:2241] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.284804 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 28e490ba5d08498f921b50f057b88369 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.308076 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c934a46a8f6c407caa7a84d6415d6901" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.401155 911 raft_consensus.cc:2466] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:36.401564 1160 tablet_replica.cc:331] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:36.401752 1160 raft_consensus.cc:2241] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.401895 1160 raft_consensus.cc:2270] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.402408 1160 ts_tablet_manager.cc:1905] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.315851 1163 tablet_bootstrap.cc:492] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.336055 890 tablet_service.cc:1515] Processing DeleteTablet for tablet f1a22d57d846414a8fa5855dc542b594 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.404827 1163 ts_tablet_manager.cc:1397] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.215s user 0.002s sys 0.002s
W20250625 07:52:36.357064 850 consensus_peers.cc:487] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:36.405210 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 48213.9us
I20250625 07:52:36.405617 1163 raft_consensus.cc:357] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.405763 1163 raft_consensus.cc:383] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.405804 1163 raft_consensus.cc:738] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.405977 1163 consensus_queue.cc:260] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.406381 1163 ts_tablet_manager.cc:1428] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250625 07:52:36.407663 985 leader_election.cc:304] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.408282 2447 raft_consensus.cc:2747] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.409701 1163 tablet_bootstrap.cc:492] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.411980 1163 tablet_bootstrap.cc:654] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.415066 1160 ts_tablet_manager.cc:1918] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:36.415194 1160 log.cc:1199] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/28e490ba5d08498f921b50f057b88369
I20250625 07:52:36.415529 1160 ts_tablet_manager.cc:1939] T 28e490ba5d08498f921b50f057b88369 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:36.418037 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]) successfully deleted
W20250625 07:52:36.418375 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f1a22d57d846414a8fa5855dc542b594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a22d57d846414a8fa5855dc542b594 already in progress: creating tablet
I20250625 07:52:36.424808 2423 raft_consensus.cc:491] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.425010 2423 raft_consensus.cc:513] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.425673 2423 leader_election.cc:290] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.427796 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.427868 2423 raft_consensus.cc:491] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.427986 2423 raft_consensus.cc:513] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.428541 2423 leader_election.cc:290] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.429394 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.429931 850 leader_election.cc:343] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.430366 851 leader_election.cc:343] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.430460 851 leader_election.cc:304] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.430804 2423 raft_consensus.cc:2747] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.431021 1163 tablet_bootstrap.cc:492] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.431139 1163 ts_tablet_manager.cc:1397] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20250625 07:52:36.431370 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c858e286fbba450e81c6cc55f0f2b613" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.431926 1163 raft_consensus.cc:357] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.431993 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c858e286fbba450e81c6cc55f0f2b613" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.432073 1163 raft_consensus.cc:383] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.432121 1163 raft_consensus.cc:738] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.432332 1163 consensus_queue.cc:260] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:36.432475 850 leader_election.cc:343] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.432937 851 leader_election.cc:343] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.433044 851 leader_election.cc:304] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.433290 2423 raft_consensus.cc:2747] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.359402 888 tablet_service.cc:1468] Processing CreateTablet for tablet b6c3420e04e2408f814fa832e8904d51 (DEFAULT_TABLE table=default.Table_d9b9ca707e724747aa3b189289a21de4 [id=0f57ae56973e4107b9c94567d6a104b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:36.442844 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c3420e04e2408f814fa832e8904d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:36.432945 1163 ts_tablet_manager.cc:1428] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:36.445320 1163 tablet_bootstrap.cc:492] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.448166 1163 tablet_bootstrap.cc:654] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.450112 713 leader_election.cc:304] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.453263 2423 raft_consensus.cc:491] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.453413 2423 raft_consensus.cc:513] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.454006 2423 leader_election.cc:290] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.454964 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea62dfb8f20f4caeaa79ea1e85d99930" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.455605 850 leader_election.cc:343] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.464092 1162 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.464329 1162 ts_tablet_manager.cc:1397] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.217s user 0.062s sys 0.066s
I20250625 07:52:36.466616 1158 tablet_bootstrap.cc:654] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.469378 1161 ts_tablet_manager.cc:1918] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.469595 1161 log.cc:1199] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/1afe90251c964e369d05e82c7db19326
I20250625 07:52:36.469973 1161 ts_tablet_manager.cc:1939] T 1afe90251c964e369d05e82c7db19326 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.470366 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1202) took 1084 ms. Trace:
I20250625 07:52:36.470517 1161 rpcz_store.cc:276] 0625 07:52:35.386095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.386145 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:36.302768 (+916623us) ts_tablet_manager.cc:1145] Deleting tablet 1afe90251c964e369d05e82c7db19326
0625 07:52:36.302778 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.357929 (+ 55151us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.469213 (+111284us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.470345 (+ 1132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":916464}
I20250625 07:52:36.471050 1161 tablet_replica.cc:331] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.471244 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 1afe90251c964e369d05e82c7db19326 (table default.Table_d14c6b22d6b745c9aadedb8f7725f4b4 [id=bca3051899e5496a8eaaa68a1b2fe4a1]) successfully deleted
I20250625 07:52:36.471335 1161 raft_consensus.cc:2241] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.471522 1161 raft_consensus.cc:2270] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.472081 1161 ts_tablet_manager.cc:1905] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.476076 2407 raft_consensus.cc:2270] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.476454 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea62dfb8f20f4caeaa79ea1e85d99930" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.476701 2407 ts_tablet_manager.cc:1905] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.477412 2161 raft_consensus.cc:2802] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 07:52:36.478921 851 leader_election.cc:343] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.478998 2161 raft_consensus.cc:695] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:36.479051 851 leader_election.cc:304] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.479249 2161 consensus_queue.cc:237] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.479454 2423 raft_consensus.cc:2747] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.480108 1162 raft_consensus.cc:357] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.480293 1162 raft_consensus.cc:383] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.480418 1162 raft_consensus.cc:738] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.480736 1162 consensus_queue.cc:260] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.481412 1162 ts_tablet_manager.cc:1428] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250625 07:52:36.481479 2045 consensus_queue.cc:1035] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.481726 1162 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.483067 1162 tablet_bootstrap.cc:654] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.484266 2423 raft_consensus.cc:491] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.484392 2423 raft_consensus.cc:513] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.485149 2423 leader_election.cc:290] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.485602 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10149f4ae0645ba88508ca51ae9f77a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.486048 850 leader_election.cc:343] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.486155 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10149f4ae0645ba88508ca51ae9f77a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.486694 851 leader_election.cc:343] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.486800 851 leader_election.cc:304] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.487243 2423 raft_consensus.cc:2747] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.488898 1163 tablet_bootstrap.cc:492] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.489181 1163 ts_tablet_manager.cc:1397] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250625 07:52:36.492566 1163 raft_consensus.cc:357] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.493836 1163 raft_consensus.cc:383] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.494212 1163 raft_consensus.cc:738] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.494627 1163 consensus_queue.cc:260] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.497175 1163 ts_tablet_manager.cc:1428] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:36.497941 1163 tablet_bootstrap.cc:492] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.505628 1163 tablet_bootstrap.cc:654] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.512830 2423 raft_consensus.cc:491] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.512990 2423 raft_consensus.cc:513] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.513622 2423 leader_election.cc:290] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.514843 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe630469fe7c4f4bac28277896eddc9c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.515436 850 leader_election.cc:343] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.515508 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe630469fe7c4f4bac28277896eddc9c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.515970 851 leader_election.cc:343] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.516069 851 leader_election.cc:304] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.516469 2423 raft_consensus.cc:2747] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.521929 2407 ts_tablet_manager.cc:1918] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.532048 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f8f803ed26a94ee081bd9a7c7eb1d660
W20250625 07:52:36.535158 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57c3d0810cf44a33b2acd50707931d26 consensus state has no leader
I20250625 07:52:36.535151 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.536446 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 113c05d81d2943e38bcd62bf83a46a2e
I20250625 07:52:36.538529 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.538714 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:36.539366 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 113c05d81d2943e38bcd62bf83a46a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113c05d81d2943e38bcd62bf83a46a2e already in progress: creating tablet
I20250625 07:52:36.541146 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:36.542558 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f8f803ed26a94ee081bd9a7c7eb1d660 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8f803ed26a94ee081bd9a7c7eb1d660 already in progress: creating tablet
I20250625 07:52:36.534008 758 tablet_service.cc:1515] Processing DeleteTablet for tablet f8f803ed26a94ee081bd9a7c7eb1d660 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.542955 2407 log.cc:1199] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:36.541183 1162 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.542495 2161 consensus_queue.cc:1035] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.543318 2407 ts_tablet_manager.cc:1939] T 0b7daf759e324c1db853fdbaab76c721 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:36.537878 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 113c05d81d2943e38bcd62bf83a46a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.543716 1162 ts_tablet_manager.cc:1397] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.009s
I20250625 07:52:36.544731 1162 raft_consensus.cc:357] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.544992 1162 raft_consensus.cc:383] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.545104 1162 raft_consensus.cc:738] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.545420 1162 consensus_queue.cc:260] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.547118 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e4cbeb746c0542c689f6853adce7a070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.548650 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4cbeb746c0542c689f6853adce7a070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.549482 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_99ec121a2e4c46baacd7c0ef2e335c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.550062 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99ec121a2e4c46baacd7c0ef2e335c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.551560 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b8f97eac18cc433b8871e352302a53c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.552093 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8f97eac18cc433b8871e352302a53c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.553862 1162 ts_tablet_manager.cc:1428] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250625 07:52:36.557792 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.558467 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.559345 1162 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.561123 1162 tablet_bootstrap.cc:654] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.561292 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 0b7daf759e324c1db853fdbaab76c721 (table default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]) successfully deleted
I20250625 07:52:36.562757 1163 tablet_bootstrap.cc:492] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.562906 1163 ts_tablet_manager.cc:1397] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.008s
I20250625 07:52:36.563719 1163 raft_consensus.cc:357] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.563901 1163 raft_consensus.cc:383] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.563957 1163 raft_consensus.cc:738] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.564178 1163 consensus_queue.cc:260] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.564734 1163 ts_tablet_manager.cc:1428] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:36.565003 1163 tablet_bootstrap.cc:492] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:36.572893 713 consensus_peers.cc:487] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e490ba5d08498f921b50f057b88369. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.573814 1163 tablet_bootstrap.cc:654] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.574119 2407 tablet_replica.cc:331] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:36.575711 2407 raft_consensus.cc:2241] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.581467 2407 raft_consensus.cc:2270] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.582131 2407 ts_tablet_manager.cc:1905] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.589675 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4] alter complete (version 1)
W20250625 07:52:36.584416 713 consensus_peers.cc:487] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.592088 2348 raft_consensus.cc:491] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.592214 2348 raft_consensus.cc:513] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:36.592790 2348 leader_election.cc:290] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.593604 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a41ffd566593499299aba5513ccb2f30" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.594069 713 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.598317 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.598574 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.598662 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.598798 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.598829 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.598598 1160 tablet_replica.cc:331] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:36.599022 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.599102 1160 raft_consensus.cc:2241] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.599193 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.599246 1160 raft_consensus.cc:2270] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:36.599391 713 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a41ffd566593499299aba5513ccb2f30
I20250625 07:52:36.599520 713 leader_election.cc:304] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.599737 2045 raft_consensus.cc:2747] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.599776 1160 ts_tablet_manager.cc:1905] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.600807 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.600955 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:36.600672 588 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:34495 (request call id 171) took 1022 ms (client timeout 1000 ms). Trace:
W20250625 07:52:36.601915 588 rpcz_store.cc:269] 0625 07:52:35.578259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.632852 (+ 54593us) service_pool.cc:225] Handling call
0625 07:52:35.761543 (+128691us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 00007feeb473b14d 00007feeb473b243 00007feeb13fe341 00007feeb8f7fd8d 00007feeb91035f8 00007feeaff4cb36 00007feeaff4c865 00007feeac73ec7e 00007feeac73db5e 00007feeac743d95 00007feeac7446bf
0625 07:52:36.600647 (+839104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:35.954017 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:35.954120 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:35.954125 (+ 5us) write_op.cc:435] Acquired schema lock
0625 07:52:35.954127 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:35.954224 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:35.954234 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:35.954239 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:35.954308 (+ 69us) tablet.cc:699] Row locks acquired
0625 07:52:35.954311 (+ 3us) write_op.cc:260] PREPARE: finished
0625 07:52:35.954345 (+ 34us) write_op.cc:270] Start()
0625 07:52:35.954373 (+ 28us) write_op.cc:276] Timestamp: P: 1750837955954279 usec, L: 0
0625 07:52:35.954377 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:35.954556 (+ 179us) log.cc:844] Serialized 356 byte log entry
0625 07:52:36.119009 (+164453us) op_driver.cc:464] REPLICATION: finished
0625 07:52:36.119393 (+ 384us) write_op.cc:301] APPLY: starting
0625 07:52:36.119412 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:36.119451 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:36.119477 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:36.119528 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:36.119563 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:36.119570 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:36.119641 (+ 71us) log.cc:844] Serialized 23 byte log entry
0625 07:52:36.119706 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.119722 (+ 16us) write_op.cc:454] Released schema lock
0625 07:52:36.119728 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:36.119803 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.119808 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0625 07:52:36.583578 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:36.583701 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:36.583706 (+ 5us) write_op.cc:435] Acquired schema lock
0625 07:52:36.583709 (+ 3us) tablet.cc:660] Decoding operations
0625 07:52:36.583822 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:36.583835 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:36.583839 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:36.583906 (+ 67us) tablet.cc:699] Row locks acquired
0625 07:52:36.583908 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:36.583945 (+ 37us) write_op.cc:270] Start()
0625 07:52:36.583978 (+ 33us) write_op.cc:276] Timestamp: P: 1750837956583746 usec, L: 0
0625 07:52:36.583981 (+ 3us) op_driver.cc:348] REPLICATION: starting
0625 07:52:36.585134 (+ 1153us) log.cc:844] Serialized 356 byte log entry
0625 07:52:36.588843 (+ 3709us) op_driver.cc:464] REPLICATION: finished
0625 07:52:36.589088 (+ 245us) write_op.cc:301] APPLY: starting
0625 07:52:36.589105 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:36.589152 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:36.589177 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:36.589234 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:36.589275 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:36.589294 (+ 19us) write_op.cc:312] APPLY: finished
0625 07:52:36.589393 (+ 99us) log.cc:844] Serialized 23 byte log entry
0625 07:52:36.589455 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.589479 (+ 24us) write_op.cc:454] Released schema lock
0625 07:52:36.589485 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":286393088,"child_traces":[["op",{"apply.queue_time_us":324,"apply.run_cpu_time_us":400,"apply.run_wall_time_us":401,"num_ops":1,"prepare.queue_time_us":185536,"prepare.run_cpu_time_us":634,"prepare.run_wall_time_us":9767,"replication_time_us":164592}],["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":456,"num_ops":1,"prepare.queue_time_us":261283,"raft.queue_time_us":40,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":279,"replication_time_us":4825}]]}
I20250625 07:52:36.603585 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.604646 1158 tablet_bootstrap.cc:492] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:36.604828 1158 ts_tablet_manager.cc:1397] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.217s user 0.007s sys 0.002s
I20250625 07:52:36.605234 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ce18b3b362544adab4b9dc33d37ae532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:36.605641 1158 raft_consensus.cc:357] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.605793 1158 raft_consensus.cc:383] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.605842 1158 raft_consensus.cc:738] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:36.605867 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce18b3b362544adab4b9dc33d37ae532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.606724 1158 consensus_queue.cc:260] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.607527 1158 ts_tablet_manager.cc:1428] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:36.607847 1158 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.609903 1158 tablet_bootstrap.cc:654] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.613080 2423 raft_consensus.cc:491] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.613232 2423 raft_consensus.cc:513] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.613879 2423 leader_election.cc:290] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.614457 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.614991 851 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.615427 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.615828 850 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.615980 850 leader_election.cc:304] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.616533 2423 raft_consensus.cc:2747] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.624837 2443 tablet.cc:1722] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:36.625331 593 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c5fe3ea5745743019d23d49ff808070a (table default.Table_4bb938fd70714708b7c8359750b8c076 [id=c48e5060ff8144d085c60a7790e55fcd]). Expected version 3 got 0
W20250625 07:52:36.625726 713 consensus_peers.cc:487] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.626211 1163 tablet_bootstrap.cc:492] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.626334 1163 ts_tablet_manager.cc:1397] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20250625 07:52:36.627108 1163 raft_consensus.cc:357] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.627297 1163 raft_consensus.cc:383] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.627351 1163 raft_consensus.cc:738] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.627526 1163 consensus_queue.cc:260] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.628002 1163 ts_tablet_manager.cc:1428] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:36.628264 1163 tablet_bootstrap.cc:492] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.634955 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ea8eb0c6e92048c482cd9244519292c9" } modify_external_catalogs: true
I20250625 07:52:36.635133 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ea8eb0c6e92048c482cd9244519292c9" } modify_external_catalogs: true
I20250625 07:52:36.641896 1162 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.642055 1162 ts_tablet_manager.cc:1397] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.012s
I20250625 07:52:36.642704 1162 raft_consensus.cc:357] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.642858 1162 raft_consensus.cc:383] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.642910 1162 raft_consensus.cc:738] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.643086 1162 consensus_queue.cc:260] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.644870 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c82b840442ec4cb4993636183812ed22
I20250625 07:52:36.645483 596 catalog_manager.cc:6652] ReplaceTablet: tablet c82b840442ec4cb4993636183812ed22 deleted and replaced by tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8
I20250625 07:52:36.648000 890 tablet_service.cc:1515] Processing DeleteTablet for tablet c82b840442ec4cb4993636183812ed22 with delete_type TABLET_DATA_DELETED (replaced by tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:36.648530 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c82b840442ec4cb4993636183812ed22 with delete_type TABLET_DATA_DELETED (replaced by tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:36.649847 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c82b840442ec4cb4993636183812ed22 with delete_type TABLET_DATA_DELETED (replaced by tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.650807 1162 ts_tablet_manager.cc:1428] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250625 07:52:36.654219 1163 tablet_bootstrap.cc:654] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.655717 1162 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.658226 1162 tablet_bootstrap.cc:654] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.667523 1161 ts_tablet_manager.cc:1918] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.667663 1161 log.cc:1199] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3ab38e0cdcd14f209e6877f2e4a46e98
I20250625 07:52:36.667999 1161 ts_tablet_manager.cc:1939] T 3ab38e0cdcd14f209e6877f2e4a46e98 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.671954 1162 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.672513 1162 ts_tablet_manager.cc:1397] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20250625 07:52:36.673112 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1201) took 1287 ms. Trace:
I20250625 07:52:36.673224 1161 rpcz_store.cc:276] 0625 07:52:35.385949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.386435 (+ 486us) service_pool.cc:225] Handling call
0625 07:52:36.470993 (+1084558us) ts_tablet_manager.cc:1145] Deleting tablet 3ab38e0cdcd14f209e6877f2e4a46e98
0625 07:52:36.471005 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.500566 (+ 29561us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.667385 (+166819us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.673094 (+ 5709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084422}
I20250625 07:52:36.673635 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1203) took 1109 ms. Trace:
I20250625 07:52:36.673712 1161 rpcz_store.cc:276] 0625 07:52:35.563814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.563868 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:36.673515 (+1109647us) ts_tablet_manager.cc:1145] Deleting tablet 08521c443c284f6aac4342b8ceb24d86
0625 07:52:36.673524 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.673618 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109479}
I20250625 07:52:36.674010 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1205) took 1002 ms. Trace:
I20250625 07:52:36.674068 1161 rpcz_store.cc:276] 0625 07:52:35.671784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.671970 (+ 186us) service_pool.cc:225] Handling call
0625 07:52:36.673928 (+1001958us) ts_tablet_manager.cc:1145] Deleting tablet 10de48baf54c40449024eadbd264ba94
0625 07:52:36.673935 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.673998 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001771}
I20250625 07:52:36.674322 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1206) took 1002 ms. Trace:
I20250625 07:52:36.674382 1161 rpcz_store.cc:276] 0625 07:52:35.671915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.671962 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:36.674237 (+1002275us) ts_tablet_manager.cc:1145] Deleting tablet 29a2f1c24e0240c0b1f0007961ee3ad2
0625 07:52:36.674245 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.674309 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19,"tablet-delete.queue_time_us":1002080}
I20250625 07:52:36.674607 1161 tablet_replica.cc:331] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.674765 1161 raft_consensus.cc:2241] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.674889 1161 raft_consensus.cc:2270] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.675366 1161 ts_tablet_manager.cc:1905] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.677891 1162 raft_consensus.cc:357] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.678129 1162 raft_consensus.cc:383] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.678221 1162 raft_consensus.cc:738] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.678493 1162 consensus_queue.cc:260] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.679386 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3ab38e0cdcd14f209e6877f2e4a46e98 (table default.Table_38b762c38ec54ae09b10d5524e53a950 [id=e634137156da455f8ff6c1271981fb6c]) successfully deleted
W20250625 07:52:36.679652 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 08521c443c284f6aac4342b8ceb24d86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08521c443c284f6aac4342b8ceb24d86 already in progress: creating tablet
W20250625 07:52:36.679827 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 10de48baf54c40449024eadbd264ba94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10de48baf54c40449024eadbd264ba94 already in progress: creating tablet
W20250625 07:52:36.680014 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 29a2f1c24e0240c0b1f0007961ee3ad2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29a2f1c24e0240c0b1f0007961ee3ad2 already in progress: creating tablet
W20250625 07:52:36.682996 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b4d308b0b16a4ab6a3a3589087c94136 (table default.Table_81dacedf9d814b60986bc865cc7cbf94 [id=c8f7f6a1e10b4a45905049e48f7cd7c7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b4d308b0b16a4ab6a3a3589087c94136 consensus state has no leader
I20250625 07:52:36.685319 1162 ts_tablet_manager.cc:1428] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20250625 07:52:36.685770 1163 tablet_bootstrap.cc:492] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.686512 1163 ts_tablet_manager.cc:1397] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250625 07:52:36.687588 1163 raft_consensus.cc:357] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.687919 1163 raft_consensus.cc:383] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.688031 1163 raft_consensus.cc:738] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.688755 1158 tablet_bootstrap.cc:492] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:36.688937 1158 ts_tablet_manager.cc:1397] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.009s
I20250625 07:52:36.689761 1158 raft_consensus.cc:357] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.689983 1158 raft_consensus.cc:383] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.690083 1158 raft_consensus.cc:738] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.690366 1158 consensus_queue.cc:260] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.692860 1158 ts_tablet_manager.cc:1428] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:36.693207 1158 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.694183 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 871b603cb92249c6b0f16c825bf46232 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.694300 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet e289ea5553ac4ff983871ece4fc72ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.694742 1158 tablet_bootstrap.cc:654] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.698580 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4bfac23414d34778bd98d79dcf630b42" } new_table_name: "default.Table_0d9011da7226470ab7282dcdbc637180" modify_external_catalogs: true
I20250625 07:52:36.701424 1162 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.701530 2161 consensus_queue.cc:1035] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.703660 1162 tablet_bootstrap.cc:654] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.711093 2161 raft_consensus.cc:491] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.711287 2161 raft_consensus.cc:513] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.712056 2161 leader_election.cc:290] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.715827 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b83f4c8544946d9adb3346df200b215 consensus state has no leader
I20250625 07:52:36.716017 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu pid 492
I20250625 07:52:36.717123 1163 consensus_queue.cc:260] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.717891 1163 ts_tablet_manager.cc:1428] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20250625 07:52:36.718231 1163 tablet_bootstrap.cc:492] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.719776 1163 tablet_bootstrap.cc:654] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.720598 1162 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.720544 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.720781 914 raft_consensus.cc:2466] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:36.720902 1162 ts_tablet_manager.cc:1397] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250625 07:52:36.721683 1162 raft_consensus.cc:357] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.721927 1162 raft_consensus.cc:383] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.722029 1162 raft_consensus.cc:738] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.722302 1162 consensus_queue.cc:260] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.722589 713 leader_election.cc:304] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:36.723017 2348 raft_consensus.cc:2802] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:36.723095 2348 raft_consensus.cc:491] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:36.723165 2348 raft_consensus.cc:3058] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:36.730507 713 consensus_peers.cc:487] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.730758 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.731194 713 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.735146 1162 ts_tablet_manager.cc:1428] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250625 07:52:36.739534 2423 consensus_queue.cc:1035] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.742003 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet f7f82e83ed924d2bbedd33d96642a10a (table default.Table_a4acbad3c3ab41a4896fd9a719d52175 [id=26371e29bd6e46ef86cd7db4c00bde58]). Expected version 3 got 0
I20250625 07:52:36.742981 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_614158d8af6a454aadfae0f0af108096"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.743548 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_614158d8af6a454aadfae0f0af108096 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.744323 1162 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.746337 1162 tablet_bootstrap.cc:654] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.746562 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250625 07:52:36.749864 1163 tablet_bootstrap.cc:492] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.750072 1163 ts_tablet_manager.cc:1397] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20250625 07:52:36.757058 1158 tablet_bootstrap.cc:492] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:36.757114 2161 consensus_queue.cc:1035] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.757294 1158 ts_tablet_manager.cc:1397] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250625 07:52:36.758153 1158 raft_consensus.cc:357] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.758318 1158 raft_consensus.cc:383] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.758368 1158 raft_consensus.cc:738] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.758594 1158 consensus_queue.cc:260] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.762315 2447 raft_consensus.cc:491] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.762558 2447 raft_consensus.cc:513] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.763358 2447 leader_election.cc:290] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:36.764191 985 leader_election.cc:343] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 871b603cb92249c6b0f16c825bf46232
W20250625 07:52:36.764533 986 leader_election.cc:343] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 871b603cb92249c6b0f16c825bf46232
I20250625 07:52:36.764649 1158 ts_tablet_manager.cc:1428] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:36.764657 986 leader_election.cc:304] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.764986 1158 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.765434 2447 raft_consensus.cc:2747] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.766758 1158 tablet_bootstrap.cc:654] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.773557 1163 raft_consensus.cc:357] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.773869 1163 raft_consensus.cc:383] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.773962 1163 raft_consensus.cc:738] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.774224 1163 consensus_queue.cc:260] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.774894 1163 ts_tablet_manager.cc:1428] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:36.775235 1163 tablet_bootstrap.cc:492] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.776178 2348 raft_consensus.cc:513] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.776885 1163 tablet_bootstrap.cc:654] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.776868 2348 leader_election.cc:290] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.777362 1160 ts_tablet_manager.cc:1918] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.777465 1160 log.cc:1199] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:36.777776 1160 ts_tablet_manager.cc:1939] T 9a90bb52358f4c38a540ad025e9e0430 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:36.780131 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:36.780404 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af43b9ceeb4f2b812ee251ea8545ce" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:36.780579 914 raft_consensus.cc:3058] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:36.783953 713 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.784370 2423 raft_consensus.cc:491] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.784521 2423 raft_consensus.cc:513] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.784788 1048 raft_consensus.cc:3058] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.785233 2423 leader_election.cc:290] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.785746 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a396bde4b5964697ad0090eab777ba0e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.786417 850 leader_election.cc:343] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.787458 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a396bde4b5964697ad0090eab777ba0e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.788266 851 leader_election.cc:343] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.788926 851 leader_election.cc:304] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.789906 2345 consensus_queue.cc:1035] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:36.790305 2423 raft_consensus.cc:2747] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.801602 1162 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.801764 1162 ts_tablet_manager.cc:1397] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.058s user 0.014s sys 0.002s
I20250625 07:52:36.802428 1162 raft_consensus.cc:357] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.802597 1162 raft_consensus.cc:383] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.802645 1162 raft_consensus.cc:738] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.803941 1162 consensus_queue.cc:260] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.805608 1158 tablet_bootstrap.cc:492] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:36.805768 1158 ts_tablet_manager.cc:1397] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250625 07:52:36.805759 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a9c6f935691440fe82ca615932a43240" } new_table_name: "default.Table_462e9992625a498e814ed911f3ee61a6" modify_external_catalogs: true
I20250625 07:52:36.806497 1162 ts_tablet_manager.cc:1428] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:36.806556 1158 raft_consensus.cc:357] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.806710 1158 raft_consensus.cc:383] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.806751 1158 raft_consensus.cc:738] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.806841 1162 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.806960 1158 consensus_queue.cc:260] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.809742 1162 tablet_bootstrap.cc:654] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.812862 1158 ts_tablet_manager.cc:1428] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:36.813175 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet b2bb060cdd9a48688967989d7491cfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.827068 1163 tablet_bootstrap.cc:492] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.827354 1163 ts_tablet_manager.cc:1397] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20250625 07:52:36.828147 1163 raft_consensus.cc:357] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:36.828378 713 consensus_peers.cc:487] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:36.828404 1163 raft_consensus.cc:383] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.828526 1163 raft_consensus.cc:738] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:36.828614 713 consensus_peers.cc:487] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:36.828814 713 consensus_peers.cc:487] T 4b4627f4da6b4542aff1e44157087b14 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:36.828996 1163 consensus_queue.cc:260] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.829697 1163 ts_tablet_manager.cc:1428] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:36.830025 1163 tablet_bootstrap.cc:492] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.831530 1163 tablet_bootstrap.cc:654] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.846412 2475 raft_consensus.cc:491] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.846591 2475 raft_consensus.cc:513] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.847209 2475 leader_election.cc:290] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.847708 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.847967 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.848394 850 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.848606 851 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.848773 851 leader_election.cc:304] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.849179 2475 raft_consensus.cc:2747] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.851070 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_656035a4978b49bd86d2e4462a67a4dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:36.851670 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_656035a4978b49bd86d2e4462a67a4dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:36.853576 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 0929ed76622248d1a41f08f4258108cd with delete_type TABLET_DATA_DELETED (replaced by tablet 5099f01a4a124d23a7c32accbd576e54) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:36.854243 2475 raft_consensus.cc:491] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.854369 2475 raft_consensus.cc:513] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.854934 2475 leader_election.cc:290] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:36.855621 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:36.856179 851 leader_election.cc:343] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.856781 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:36.857292 850 leader_election.cc:343] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.857766 850 leader_election.cc:304] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.858016 2475 raft_consensus.cc:2747] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.861582 1163 tablet_bootstrap.cc:492] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.861925 1163 ts_tablet_manager.cc:1397] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.001s
I20250625 07:52:36.862797 1163 raft_consensus.cc:357] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.863021 1163 raft_consensus.cc:383] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.863114 1163 raft_consensus.cc:738] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.863404 1163 consensus_queue.cc:260] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.863517 2348 raft_consensus.cc:491] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.863727 2348 raft_consensus.cc:513] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.864028 1163 ts_tablet_manager.cc:1428] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:36.864418 2348 leader_election.cc:290] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.866693 914 raft_consensus.cc:2466] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:36.868328 1160 tablet_replica.cc:331] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:36.868654 1160 raft_consensus.cc:2241] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.868871 1160 raft_consensus.cc:2270] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.869884 1160 ts_tablet_manager.cc:1905] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.871435 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9a90bb52358f4c38a540ad025e9e0430 (table default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]) successfully deleted
I20250625 07:52:36.873754 1161 ts_tablet_manager.cc:1918] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.873878 1161 log.cc:1199] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7480c07af7ce4a3889bd8d42d7419d57
I20250625 07:52:36.874171 1161 ts_tablet_manager.cc:1939] T 7480c07af7ce4a3889bd8d42d7419d57 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:36.876571 1163 tablet_bootstrap.cc:492] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:36.876551 1162 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.878439 1163 tablet_bootstrap.cc:654] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.880788 1162 ts_tablet_manager.cc:1397] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.035s
I20250625 07:52:36.813176 1158 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:36.881609 1162 raft_consensus.cc:357] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.881810 1162 raft_consensus.cc:383] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.881862 1162 raft_consensus.cc:738] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.882100 1162 consensus_queue.cc:260] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.882630 1162 ts_tablet_manager.cc:1428] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:36.882951 1162 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.884186 1158 tablet_bootstrap.cc:654] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.896544 2348 raft_consensus.cc:491] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.896787 2348 raft_consensus.cc:513] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.897588 2348 leader_election.cc:290] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.902511 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.903522 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.903721 914 raft_consensus.cc:2466] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:36.906327 2443 tablet.cc:1722] T c934a46a8f6c407caa7a84d6415d6901 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:36.884651 1162 tablet_bootstrap.cc:654] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.919198 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:36.927675 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.927896 914 raft_consensus.cc:2466] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:36.929052 713 leader_election.cc:304] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:36.929419 2161 raft_consensus.cc:2802] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:36.929544 2161 raft_consensus.cc:695] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:36.929801 2161 consensus_queue.cc:237] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.933655 1162 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:36.933795 1162 ts_tablet_manager.cc:1397] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20250625 07:52:36.934507 1162 raft_consensus.cc:357] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.934669 1162 raft_consensus.cc:383] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.934717 1162 raft_consensus.cc:738] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.934906 1162 consensus_queue.cc:260] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.935456 1162 ts_tablet_manager.cc:1428] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:36.935724 1162 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:36.937541 1162 tablet_bootstrap.cc:654] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:36.939613 2407 ts_tablet_manager.cc:1918] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:36.941931 2407 log.cc:1199] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:36.948441 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1209) took 1187 ms. Trace:
I20250625 07:52:36.948570 1161 rpcz_store.cc:276] 0625 07:52:35.761051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.761956 (+ 905us) service_pool.cc:225] Handling call
0625 07:52:36.674571 (+912615us) ts_tablet_manager.cc:1145] Deleting tablet 7480c07af7ce4a3889bd8d42d7419d57
0625 07:52:36.674579 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.771965 (+ 97386us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.873612 (+101647us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:36.948409 (+ 74797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":912444}
I20250625 07:52:36.949132 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1210) took 1172 ms. Trace:
I20250625 07:52:36.949227 1161 rpcz_store.cc:276] 0625 07:52:35.777105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.777175 (+ 70us) service_pool.cc:225] Handling call
0625 07:52:36.948976 (+1171801us) ts_tablet_manager.cc:1145] Deleting tablet d74924e8dee943e0ae4c93f0db1d75f4
0625 07:52:36.948987 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.949111 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171582}
I20250625 07:52:36.949486 1161 tablet_replica.cc:331] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:36.949663 1161 raft_consensus.cc:2241] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.949820 1161 raft_consensus.cc:2270] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.950413 1161 ts_tablet_manager.cc:1905] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.957343 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7480c07af7ce4a3889bd8d42d7419d57 (table default.Table_9d70fa7bc97b40a2a51d2fa6e7d1e01c [id=7949cc53977a4f688c1cd6ba4d29d5eb]) successfully deleted
W20250625 07:52:36.958016 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d74924e8dee943e0ae4c93f0db1d75f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d74924e8dee943e0ae4c93f0db1d75f4 already in progress: creating tablet
I20250625 07:52:36.964859 2447 raft_consensus.cc:491] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:36.965018 2447 raft_consensus.cc:513] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.965690 2447 leader_election.cc:290] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.966320 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:36.966326 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7f82e83ed924d2bbedd33d96642a10a" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:36.969604 986 leader_election.cc:304] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:36.970129 2447 raft_consensus.cc:2747] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:36.941530 713 leader_election.cc:304] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:36.970779 2161 raft_consensus.cc:2802] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:36.970961 2161 raft_consensus.cc:491] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:36.971093 2161 raft_consensus.cc:3058] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:36.972050 713 leader_election.cc:304] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:36.972465 2045 raft_consensus.cc:2802] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:36.972573 2045 raft_consensus.cc:491] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:36.972647 2045 raft_consensus.cc:3058] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:36.974304 713 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:36.974848 713 leader_election.cc:343] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:36.983474 2161 raft_consensus.cc:513] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.983568 2045 raft_consensus.cc:513] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.984416 2045 leader_election.cc:290] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:36.984982 2407 ts_tablet_manager.cc:1939] T 07a07af928ff4d3d894b074ef0cf44eb P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:36.985941 2407 tablet_replica.cc:331] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:36.986347 2407 raft_consensus.cc:2241] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:36.986562 2407 raft_consensus.cc:2270] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:36.987212 2407 ts_tablet_manager.cc:1905] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:36.990484 1163 tablet_bootstrap.cc:492] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:36.990722 1163 ts_tablet_manager.cc:1397] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.003s
I20250625 07:52:36.991580 1163 raft_consensus.cc:357] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:36.991839 1163 raft_consensus.cc:383] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:36.991927 1163 raft_consensus.cc:738] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:36.992228 1163 consensus_queue.cc:260] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.001574 1158 tablet_bootstrap.cc:492] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.001871 1158 ts_tablet_manager.cc:1397] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.189s user 0.004s sys 0.000s
I20250625 07:52:37.002887 1158 raft_consensus.cc:357] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.003856 1160 ts_tablet_manager.cc:1918] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.003985 1160 log.cc:1199] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:37.004086 1158 raft_consensus.cc:383] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.004274 1160 ts_tablet_manager.cc:1939] T 26d2bb8b0a8f45079083f2b42bc25011 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.005154 1158 raft_consensus.cc:738] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.005748 1158 consensus_queue.cc:260] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.006667 1158 ts_tablet_manager.cc:1428] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:37.008581 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet bb03e70e9aa740f18d8cd0543f730f83
I20250625 07:52:37.009109 1158 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.010370 890 tablet_service.cc:1515] Processing DeleteTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.012135 589 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c934a46a8f6c407caa7a84d6415d6901 (table default.Table_70dce4df11fe47c8b1ea0003e745f82d [id=6889b79d2a004cf894a70fbe2402f7b4]). Expected version 1 got 0
I20250625 07:52:37.012708 2475 raft_consensus.cc:491] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.012889 2475 raft_consensus.cc:513] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.013289 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_33cd2aff4e6b46b7a1aa026c219cd20b" } new_table_name: "default.Table_76d76b6e4e9d4ab58006f74a5aa94477" modify_external_catalogs: true
I20250625 07:52:37.013640 2475 leader_election.cc:290] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.015018 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6caaf2d07d6b4d1585502707cea12756" } modify_external_catalogs: true
I20250625 07:52:37.015172 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6caaf2d07d6b4d1585502707cea12756" } modify_external_catalogs: true
I20250625 07:52:37.015605 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "486463a8140c4e769cc79870d2acff78" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:37.015779 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a59e4843e6594fc8bb82fe450bb820cc" } modify_external_catalogs: true
I20250625 07:52:37.015915 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a59e4843e6594fc8bb82fe450bb820cc" } modify_external_catalogs: true
W20250625 07:52:37.016237 850 leader_election.cc:343] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.016439 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5c6ea2b73e764d6da9c7b6ed2068a56a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.017186 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c6ea2b73e764d6da9c7b6ed2068a56a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.018524 1158 tablet_bootstrap.cc:654] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.020851 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.023373 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 26d2bb8b0a8f45079083f2b42bc25011 (table default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]) successfully deleted
I20250625 07:52:37.023741 1160 tablet_replica.cc:331] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:37.026999 1161 ts_tablet_manager.cc:1918] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.028398 1161 log.cc:1199] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/cd728ffc2a274deea3fad533d1eaa6da
I20250625 07:52:37.028848 1161 ts_tablet_manager.cc:1939] T cd728ffc2a274deea3fad533d1eaa6da P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:37.029242 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1211) took 1244 ms. Trace:
I20250625 07:52:37.029381 1161 rpcz_store.cc:276] 0625 07:52:35.784524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.784585 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:36.949442 (+1164857us) ts_tablet_manager.cc:1145] Deleting tablet cd728ffc2a274deea3fad533d1eaa6da
0625 07:52:36.949450 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:36.954610 (+ 5160us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.026885 (+ 72275us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.029226 (+ 2341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164478}
I20250625 07:52:37.029807 1161 tablet_replica.cc:331] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:37.029879 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet cd728ffc2a274deea3fad533d1eaa6da (table default.Table_98fc1c8ebdd141a3b0500bab94206f7b [id=ca9ba1ea381c4ba6b1a472abdb298056]) successfully deleted
I20250625 07:52:37.030115 1161 raft_consensus.cc:2241] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.030331 1161 raft_consensus.cc:2270] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.030907 1161 ts_tablet_manager.cc:1905] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.035015 1158 tablet_bootstrap.cc:492] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.035360 1158 ts_tablet_manager.cc:1397] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20250625 07:52:37.036423 1158 raft_consensus.cc:357] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.036793 1158 raft_consensus.cc:383] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.036923 1158 raft_consensus.cc:738] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.037302 1158 consensus_queue.cc:260] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.038163 1158 ts_tablet_manager.cc:1428] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:37.038597 1158 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.044524 1158 tablet_bootstrap.cc:654] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.046813 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7b5cc617ff6f4d078c29187d9c38846d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:37.046999 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet a1baf94a051e485eaf4c0e074e147252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:37.048722 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b5cc617ff6f4d078c29187d9c38846d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.053684 1163 ts_tablet_manager.cc:1428] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.063s user 0.000s sys 0.001s
I20250625 07:52:37.054133 1163 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:37.055858 1163 tablet_bootstrap.cc:654] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.056108 1160 raft_consensus.cc:2241] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.057539 1160 raft_consensus.cc:2270] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.058271 1160 ts_tablet_manager.cc:1905] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.068248 1158 tablet_bootstrap.cc:492] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.068534 1158 ts_tablet_manager.cc:1397] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20250625 07:52:37.069672 1158 raft_consensus.cc:357] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.069979 1158 raft_consensus.cc:383] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.070111 1158 raft_consensus.cc:738] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.070520 1158 consensus_queue.cc:260] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.077180 1158 ts_tablet_manager.cc:1428] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:37.079639 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_37ecb82090564313862912776a682691" } modify_external_catalogs: true
I20250625 07:52:37.079836 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_37ecb82090564313862912776a682691" } modify_external_catalogs: true
I20250625 07:52:37.081233 1158 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.086505 1158 tablet_bootstrap.cc:654] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.087664 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e4425d3b61cf47d58737fcf7a4dcac7f" } new_table_name: "default.Table_38d929aa141642c88be60057a30a40f8" modify_external_catalogs: true
I20250625 07:52:37.087826 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 019959e9abed46fbbbb4755cb01cef44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:37.094806 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8831e56790584f5b8c77f02b3e8add0a has no consensus state
I20250625 07:52:37.095160 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb" } modify_external_catalogs: true
I20250625 07:52:37.095314 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9549f696d1fe4fd38b64ce3b5eaf3dbb" } modify_external_catalogs: true
I20250625 07:52:37.096328 2348 raft_consensus.cc:491] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.097810 2045 raft_consensus.cc:491] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.097959 2045 raft_consensus.cc:513] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.098572 593 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b3c54204d5446c688cb123edbd430c1 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:37.098685 2288 raft_consensus.cc:491] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.098822 2288 raft_consensus.cc:513] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:37.099327 2348 raft_consensus.cc:513] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.099766 2348 leader_election.cc:290] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.101701 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_37c181e25508461b927d103f11855cf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.102596 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37c181e25508461b927d103f11855cf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.102843 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:37.103626 1158 tablet_bootstrap.cc:492] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.103782 1158 ts_tablet_manager.cc:1397] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250625 07:52:37.103790 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32a9f4cf6838411fab06fa6d35ae9492" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:37.104010 914 raft_consensus.cc:3058] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:37.104543 1158 raft_consensus.cc:357] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.104744 1158 raft_consensus.cc:383] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.104792 1158 raft_consensus.cc:738] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.105005 1158 consensus_queue.cc:260] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.107244 2475 raft_consensus.cc:491] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.107409 2475 raft_consensus.cc:513] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.108148 2475 leader_election.cc:290] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.109763 914 raft_consensus.cc:2466] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:37.110507 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08cd482ceec244c2bed2b116d5c14d89" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:37.110778 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 07a07af928ff4d3d894b074ef0cf44eb (table default.Table_eb85d0d01edc400eabcdd8a8175dfbf9 [id=68ab97279148448991b34aeef51de39a]) successfully deleted
I20250625 07:52:37.110915 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08cd482ceec244c2bed2b116d5c14d89" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:37.111416 851 leader_election.cc:343] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.111717 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:37.112053 850 leader_election.cc:343] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.112162 850 leader_election.cc:304] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.112557 2475 raft_consensus.cc:2747] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.113260 1158 ts_tablet_manager.cc:1428] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 07:52:37.113600 1158 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.115299 1158 tablet_bootstrap.cc:654] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.115661 752 tablet_service.cc:1515] Processing DeleteTablet for tablet bb03e70e9aa740f18d8cd0543f730f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:36 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.118036 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "486463a8140c4e769cc79870d2acff78" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:37.124938 851 leader_election.cc:343] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.125084 851 leader_election.cc:304] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.125597 2475 raft_consensus.cc:2747] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.125908 2288 leader_election.cc:290] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.126240 1158 tablet_bootstrap.cc:492] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.126423 1158 ts_tablet_manager.cc:1397] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250625 07:52:37.126904 2045 leader_election.cc:290] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.127610 2161 leader_election.cc:290] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.128417 1158 raft_consensus.cc:357] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.128733 1158 raft_consensus.cc:383] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.128834 1158 raft_consensus.cc:738] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.129110 1158 consensus_queue.cc:260] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.132447 1158 ts_tablet_manager.cc:1428] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250625 07:52:37.132851 1158 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.133420 1162 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:37.133631 1162 ts_tablet_manager.cc:1397] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.198s user 0.023s sys 0.027s
I20250625 07:52:37.134541 1162 raft_consensus.cc:357] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.134778 1162 raft_consensus.cc:383] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.134871 1162 raft_consensus.cc:738] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.135138 1162 consensus_queue.cc:260] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.135787 1162 ts_tablet_manager.cc:1428] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:37.136154 1162 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:37.137313 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70094e6c7e644f62bf5ef04dcfb4c380" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:37.138907 1158 tablet_bootstrap.cc:654] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.141131 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f47caf3bae41b79b0aa1e320f15da4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:37.141947 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f47caf3bae41b79b0aa1e320f15da4" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:37.143764 713 leader_election.cc:304] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:37.144240 713 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:37.144471 2494 raft_consensus.cc:2802] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
W20250625 07:52:37.144533 713 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Illegal state: Tablet not RUNNING: STOPPED
I20250625 07:52:37.144627 2494 raft_consensus.cc:695] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:37.144915 2494 consensus_queue.cc:237] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.147235 1162 tablet_bootstrap.cc:654] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.153244 1160 ts_tablet_manager.cc:1918] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.171614 2407 ts_tablet_manager.cc:1918] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:37.172237 713 leader_election.cc:343] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:37.172828 713 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:37.173441 713 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:37.173723 713 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.173878 713 leader_election.cc:304] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.174194 2045 raft_consensus.cc:2747] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.174777 2497 tablet.cc:1722] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:37.181707 1158 tablet_bootstrap.cc:492] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.181982 1158 ts_tablet_manager.cc:1397] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.002s
I20250625 07:52:37.183746 1158 raft_consensus.cc:357] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.184207 1158 raft_consensus.cc:383] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.184312 1158 raft_consensus.cc:738] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.184588 1158 consensus_queue.cc:260] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.188086 1158 ts_tablet_manager.cc:1428] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:37.188478 1158 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.189795 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8b3c54204d5446c688cb123edbd430c1
I20250625 07:52:37.190218 593 catalog_manager.cc:6652] ReplaceTablet: tablet 8b3c54204d5446c688cb123edbd430c1 deleted and replaced by tablet acd7c3ddfe9d4b8dbed4ed75a604205c
I20250625 07:52:37.190332 1158 tablet_bootstrap.cc:654] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.193892 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91]). Expected version 2 got 0
I20250625 07:52:37.194180 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d7ed91509ba94562a9207c7d7c0d3e2b (table default.Table_ca90d59a48d947778cd899359e52ed41 [id=1353496ce19c4b14bdcc41400f3e9844]). Expected version 1 got 0
I20250625 07:52:37.194942 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:37.195173 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:37.195537 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 6a084dfb44c7401b8958591d72e859d7 (DEFAULT_TABLE table=default.Table_8c8565ce30804a4f8148c11e83c7cce9 [id=1910c68dc2634716adc19ab233fd2802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.196285 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 44a923432f03430a8f7f65d955cabc30 (DEFAULT_TABLE table=default.Table_893afa8db5b64a3fb5a0bcbea93d04aa [id=946f6292655749119643bfa586eaf8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.196591 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:37.196590 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c33c1c9dc1b24e40bfb930fa47e95c97 (DEFAULT_TABLE table=default.Table_1c2cf30b339244f39bef84e1af811bdd [id=ed79b223049040d4a857184b67673975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.196748 1015 tablet_service.cc:1468] Processing CreateTablet for tablet c99326dc959c4032a207f85ff426b862 (DEFAULT_TABLE table=default.Table_9a0104b93e9a4a79b5140e24b91a74f5 [id=77bbac979d1c4612af0445c9d568f64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.196892 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:37.197036 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3c54204d5446c688cb123edbd430c1 with delete_type TABLET_DATA_DELETED (replaced by tablet acd7c3ddfe9d4b8dbed4ed75a604205c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.197190 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_4bfac23414d34778bd98d79dcf630b42 [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 1 got 0
W20250625 07:52:37.197574 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61e076870efc4744b1224ddcb0eef3fa consensus state has no leader
I20250625 07:52:37.197501 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f2909c29db4c434c9db393c12ea48a1d (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.197839 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 0139149bbf574e13be3b0411a83a0593 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:37.197933 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 35ec542f480f4b32b59b364f3194e988 (DEFAULT_TABLE table=default.Table_6157edbcfec44ec5a0e7ccd884de1ab7 [id=794225dd6afe4705baffd9b383440638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.198329 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531 (DEFAULT_TABLE table=default.Table_aaf276d4a7fd405d964f4808679f5fb1 [id=3623e3d46ee14bb99ec8deaa621cce78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.198691 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 35f819ef1fa74982a50d6714de305535 (DEFAULT_TABLE table=default.Table_a737026476024d37af115083c4585882 [id=b9d4f314564a43d687a82f8a5706f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.199071 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2a353629294eab9a2a27411699f795 (DEFAULT_TABLE table=default.Table_2e441da0a6b34a83a84efaa1c2041ef2 [id=7b97175ead54406d9b940e1c29bf00f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.199918 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a084dfb44c7401b8958591d72e859d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.200139 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_a23788a3c209434db9f0ef44e0220672 [id=c80e78f1ef124af2b8f405e7d162ade8] alter complete (version 2)
I20250625 07:52:37.201179 1012 tablet_service.cc:1468] Processing CreateTablet for tablet d8017c111cfd465290053a75955752d3 (DEFAULT_TABLE table=default.Table_3e1f71864f5a42d9a26edb7293a99df0 [id=0d93f6a3657547e9907edbf8a3160506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.202790 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 9bcf9bdcdf294e979ace72f4fba236be (DEFAULT_TABLE table=default.Table_ff6a95b9c737428fac9ea3ef1a5e8ffd [id=bca4de7f0a564b39833ed5fc5cc88d36]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.205554 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 998f5f465cc64052be2d32f639ab4705 consensus state has no leader
W20250625 07:52:37.208084 591 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:33657 (request call id 227) took 1145 ms (client timeout 1000 ms). Trace:
I20250625 07:52:37.209519 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33c1c9dc1b24e40bfb930fa47e95c97. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.208253 591 rpcz_store.cc:269] 0625 07:52:36.062459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.122325 (+ 59866us) service_pool.cc:225] Handling call
0625 07:52:36.146905 (+ 24580us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 00007feeb473b14d 00007feeb473b243 00007feeb13fe341 00007feeb8f7fd8d 00007feeb91035f8 00007feeaff4cb36 00007feeaff4c865 00007feeac73ec7e 00007feeac73db5e 00007feeac743d95 00007feeac7446bf
0625 07:52:37.208062 (+1061157us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:36.608122 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:36.608258 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:36.608263 (+ 5us) write_op.cc:435] Acquired schema lock
0625 07:52:36.608265 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:36.608524 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:36.608538 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:36.608543 (+ 5us) tablet.cc:683] Acquiring locks for 7 operations
0625 07:52:36.610930 (+ 2387us) tablet.cc:699] Row locks acquired
0625 07:52:36.610936 (+ 6us) write_op.cc:260] PREPARE: finished
0625 07:52:36.610983 (+ 47us) write_op.cc:270] Start()
0625 07:52:36.611030 (+ 47us) write_op.cc:276] Timestamp: P: 1750837956610782 usec, L: 0
0625 07:52:36.611037 (+ 7us) op_driver.cc:348] REPLICATION: starting
0625 07:52:36.611294 (+ 257us) log.cc:844] Serialized 2772 byte log entry
0625 07:52:36.622486 (+ 11192us) op_driver.cc:464] REPLICATION: finished
0625 07:52:36.622676 (+ 190us) write_op.cc:301] APPLY: starting
0625 07:52:36.622691 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:36.622740 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:36.622765 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:36.622886 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:36.622923 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0625 07:52:36.622930 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:36.623025 (+ 95us) log.cc:844] Serialized 59 byte log entry
0625 07:52:36.623087 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.623111 (+ 24us) write_op.cc:454] Released schema lock
0625 07:52:36.623117 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:36.636726 (+ 13609us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:36.636733 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0625 07:52:36.858080 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:36.858174 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:36.858179 (+ 5us) write_op.cc:435] Acquired schema lock
0625 07:52:36.858181 (+ 2us) tablet.cc:660] Decoding operations
0625 07:52:36.858258 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:36.858266 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:36.858270 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:36.858327 (+ 57us) tablet.cc:699] Row locks acquired
0625 07:52:36.858331 (+ 4us) write_op.cc:260] PREPARE: finished
0625 07:52:36.858422 (+ 91us) write_op.cc:270] Start()
0625 07:52:36.858458 (+ 36us) write_op.cc:276] Timestamp: P: 1750837956858212 usec, L: 0
0625 07:52:36.858462 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:36.858619 (+ 157us) log.cc:844] Serialized 356 byte log entry
0625 07:52:37.006877 (+148258us) op_driver.cc:464] REPLICATION: finished
0625 07:52:37.007004 (+ 127us) write_op.cc:301] APPLY: starting
0625 07:52:37.007016 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:37.007049 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:37.007073 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:37.007120 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:37.007147 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:37.007154 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:37.007254 (+ 100us) log.cc:844] Serialized 23 byte log entry
0625 07:52:37.007304 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:37.007318 (+ 14us) write_op.cc:454] Released schema lock
0625 07:52:37.007323 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":40,"spinlock_wait_cycles":53721344,"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":501,"apply.run_wall_time_us":501,"num_ops":7,"prepare.queue_time_us":284800,"prepare.run_cpu_time_us":1188,"prepare.run_wall_time_us":28560,"raft.queue_time_us":54,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":313,"replication_time_us":11417}],["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":371,"num_ops":1,"prepare.queue_time_us":233014,"prepare.run_cpu_time_us":555,"prepare.run_wall_time_us":837,"replication_time_us":148390}]]}
I20250625 07:52:37.212991 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8017c111cfd465290053a75955752d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.217000 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcf9bdcdf294e979ace72f4fba236be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.221091 758 tablet_service.cc:1468] Processing CreateTablet for tablet c99326dc959c4032a207f85ff426b862 (DEFAULT_TABLE table=default.Table_9a0104b93e9a4a79b5140e24b91a74f5 [id=77bbac979d1c4612af0445c9d568f64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.221498 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99326dc959c4032a207f85ff426b862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.221710 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bc8f47541c54c4287ebb7b892daa1be" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:37.222720 749 tablet_service.cc:1468] Processing CreateTablet for tablet f2909c29db4c434c9db393c12ea48a1d (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.223083 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2909c29db4c434c9db393c12ea48a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.224809 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3c54204d5446c688cb123edbd430c1 with delete_type TABLET_DATA_DELETED (replaced by tablet acd7c3ddfe9d4b8dbed4ed75a604205c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.225258 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d89fc1a546474a68951fd21e668f8d32 (DEFAULT_TABLE table=default.Table_b3a3855c16d64eeabd95e1bd8de74d2c [id=1df39597ee8945ca8171f0bf64c01960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.225610 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89fc1a546474a68951fd21e668f8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.227427 1160 log.cc:1199] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/84f47caf3bae41b79b0aa1e320f15da4
I20250625 07:52:37.161342 1163 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:37.193332 890 tablet_service.cc:1468] Processing CreateTablet for tablet 44a923432f03430a8f7f65d955cabc30 (DEFAULT_TABLE table=default.Table_893afa8db5b64a3fb5a0bcbea93d04aa [id=946f6292655749119643bfa586eaf8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.227860 1163 ts_tablet_manager.cc:1397] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.174s user 0.003s sys 0.000s
I20250625 07:52:37.221043 754 tablet_service.cc:1468] Processing CreateTablet for tablet 44a923432f03430a8f7f65d955cabc30 (DEFAULT_TABLE table=default.Table_893afa8db5b64a3fb5a0bcbea93d04aa [id=946f6292655749119643bfa586eaf8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.228179 1160 ts_tablet_manager.cc:1939] T 84f47caf3bae41b79b0aa1e320f15da4 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.228473 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a923432f03430a8f7f65d955cabc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.228713 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab27fe375d447f927312b50342e916 (DEFAULT_TABLE table=default.Table_a0e57062ab164088b1a7cf52ffe4dd18 [id=e1092aeef5b44b91a150aaa1413fbf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.229039 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab27fe375d447f927312b50342e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.231105 2407 log.cc:1199] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c263b671be384cd097041c60306034de
I20250625 07:52:37.231514 2407 ts_tablet_manager.cc:1939] T c263b671be384cd097041c60306034de P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:37.232437 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1310) took 1094 ms. Trace:
I20250625 07:52:37.232991 1023 tablet_service.cc:1468] Processing CreateTablet for tablet cff61e3f37dc4c7b99a0aae816db1f44 (DEFAULT_TABLE table=default.Table_ae0c597df57144d3b0b8fad6aacfde2f [id=01a61afc29f643ebb926991bf3849bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.233327 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff61e3f37dc4c7b99a0aae816db1f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.234869 1162 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:37.235018 1162 ts_tablet_manager.cc:1397] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20250625 07:52:37.235785 1162 raft_consensus.cc:357] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.236189 1162 raft_consensus.cc:383] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.236307 1162 raft_consensus.cc:738] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.216600 885 tablet_service.cc:1468] Processing CreateTablet for tablet c99326dc959c4032a207f85ff426b862 (DEFAULT_TABLE table=default.Table_9a0104b93e9a4a79b5140e24b91a74f5 [id=77bbac979d1c4612af0445c9d568f64d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.217221 893 tablet_service.cc:1468] Processing CreateTablet for tablet f2909c29db4c434c9db393c12ea48a1d (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.236557 1162 consensus_queue.cc:260] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.236886 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.236896 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2909c29db4c434c9db393c12ea48a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.237076 1162 ts_tablet_manager.cc:1428] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:37.237353 1162 tablet_bootstrap.cc:492] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:37.216876 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b3c54204d5446c688cb123edbd430c1 with delete_type TABLET_DATA_DELETED (replaced by tablet acd7c3ddfe9d4b8dbed4ed75a604205c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.219127 880 tablet_service.cc:1468] Processing CreateTablet for tablet 35f819ef1fa74982a50d6714de305535 (DEFAULT_TABLE table=default.Table_a737026476024d37af115083c4585882 [id=b9d4f314564a43d687a82f8a5706f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.238209 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f819ef1fa74982a50d6714de305535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.238755 894 tablet_service.cc:1468] Processing CreateTablet for tablet 6a084dfb44c7401b8958591d72e859d7 (DEFAULT_TABLE table=default.Table_8c8565ce30804a4f8148c11e83c7cce9 [id=1910c68dc2634716adc19ab233fd2802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.239049 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a084dfb44c7401b8958591d72e859d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.221132 895 tablet_service.cc:1468] Processing CreateTablet for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531 (DEFAULT_TABLE table=default.Table_aaf276d4a7fd405d964f4808679f5fb1 [id=3623e3d46ee14bb99ec8deaa621cce78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.240935 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.240994 879 tablet_service.cc:1468] Processing CreateTablet for tablet c33c1c9dc1b24e40bfb930fa47e95c97 (DEFAULT_TABLE table=default.Table_1c2cf30b339244f39bef84e1af811bdd [id=ed79b223049040d4a857184b67673975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.241587 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f819ef1fa74982a50d6714de305535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.242677 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2909c29db4c434c9db393c12ea48a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.243440 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec542f480f4b32b59b364f3194e988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.244053 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2a353629294eab9a2a27411699f795 (DEFAULT_TABLE table=default.Table_2e441da0a6b34a83a84efaa1c2041ef2 [id=7b97175ead54406d9b940e1c29bf00f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.244160 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99326dc959c4032a207f85ff426b862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.244844 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a923432f03430a8f7f65d955cabc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.246909 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2a353629294eab9a2a27411699f795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.247751 888 tablet_service.cc:1468] Processing CreateTablet for tablet 35ec542f480f4b32b59b364f3194e988 (DEFAULT_TABLE table=default.Table_6157edbcfec44ec5a0e7ccd884de1ab7 [id=794225dd6afe4705baffd9b383440638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.248293 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2bcfe7c3c54741fbb0aca69650fc8eee" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:37.248332 882 tablet_service.cc:1468] Processing CreateTablet for tablet e07b663348454158a7398cfbc34444f4 (DEFAULT_TABLE table=default.Table_747b0d5eb8ca4f1ab30aced4c1431e9c [id=9821fb2d04fd4a3896059454afa6cc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.248436 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99326dc959c4032a207f85ff426b862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.248474 909 raft_consensus.cc:3058] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:37.248618 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b663348454158a7398cfbc34444f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.248916 892 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab27fe375d447f927312b50342e916 (DEFAULT_TABLE table=default.Table_a0e57062ab164088b1a7cf52ffe4dd18 [id=e1092aeef5b44b91a150aaa1413fbf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.249081 891 tablet_service.cc:1468] Processing CreateTablet for tablet d8017c111cfd465290053a75955752d3 (DEFAULT_TABLE table=default.Table_3e1f71864f5a42d9a26edb7293a99df0 [id=0d93f6a3657547e9907edbf8a3160506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.249258 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab27fe375d447f927312b50342e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.249529 877 tablet_service.cc:1468] Processing CreateTablet for tablet 9bcf9bdcdf294e979ace72f4fba236be (DEFAULT_TABLE table=default.Table_ff6a95b9c737428fac9ea3ef1a5e8ffd [id=bca4de7f0a564b39833ed5fc5cc88d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.249797 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcf9bdcdf294e979ace72f4fba236be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.251048 909 raft_consensus.cc:2466] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:37.251753 2447 raft_consensus.cc:491] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.251868 2447 raft_consensus.cc:513] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.252403 2447 leader_election.cc:290] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.252892 876 tablet_service.cc:1468] Processing CreateTablet for tablet d89fc1a546474a68951fd21e668f8d32 (DEFAULT_TABLE table=default.Table_b3a3855c16d64eeabd95e1bd8de74d2c [id=1df39597ee8945ca8171f0bf64c01960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.253058 889 tablet_service.cc:1468] Processing CreateTablet for tablet dd8fd11ecc8c4602a682a19ea7d46c1c (DEFAULT_TABLE table=default.Table_8d936848b9fc4ef384749a5a96ce3371 [id=cb0bc1e159de48c3b50b2cfc16e9cd47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.253202 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89fc1a546474a68951fd21e668f8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.253377 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8fd11ecc8c4602a682a19ea7d46c1c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.253378 986 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:37.253530 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70094e6c7e644f62bf5ef04dcfb4c380" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:37.253074 884 tablet_service.cc:1468] Processing CreateTablet for tablet 70a104d01e3b4115a8d68d59c5a371a4 (DEFAULT_TABLE table=default.Table_6db07140365b4f2a8c6d6aaaf164be23 [id=25b23cd32a0a49e4a75708067548f761]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.253672 911 raft_consensus.cc:2466] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:37.253818 887 tablet_service.cc:1468] Processing CreateTablet for tablet 89fbdfddfc414e93898fe288b8fecb1e (DEFAULT_TABLE table=default.Table_47d9a728e38f49cf948b3bdf5ff09621 [id=c6395301fc0d4690b713be9f9cebb58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.252892 881 tablet_service.cc:1468] Processing CreateTablet for tablet cff61e3f37dc4c7b99a0aae816db1f44 (DEFAULT_TABLE table=default.Table_ae0c597df57144d3b0b8fad6aacfde2f [id=01a61afc29f643ebb926991bf3849bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.253831 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a923432f03430a8f7f65d955cabc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.252912 883 tablet_service.cc:1468] Processing CreateTablet for tablet 0b83f1c3bc02423a9dabd58555dd997b (DEFAULT_TABLE table=default.Table_22c365283ad046a7bf006c4f05f78492 [id=3fdf91140bb74aefbbd065c0aa0bf80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.254729 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec542f480f4b32b59b364f3194e988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.255002 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b83f1c3bc02423a9dabd58555dd997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.255573 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2a353629294eab9a2a27411699f795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.256335 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8017c111cfd465290053a75955752d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.256398 1019 tablet_service.cc:1468] Processing CreateTablet for tablet dd8fd11ecc8c4602a682a19ea7d46c1c (DEFAULT_TABLE table=default.Table_8d936848b9fc4ef384749a5a96ce3371 [id=cb0bc1e159de48c3b50b2cfc16e9cd47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.256775 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8fd11ecc8c4602a682a19ea7d46c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.256779 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff61e3f37dc4c7b99a0aae816db1f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.257157 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fbdfddfc414e93898fe288b8fecb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.257449 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33c1c9dc1b24e40bfb930fa47e95c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.259498 895 tablet_service.cc:1468] Processing CreateTablet for tablet b70dffecdbde4a4eb6f6951187bfd499 (DEFAULT_TABLE table=default.Table_976b8466313948adb4c34681120d4eac [id=447e050016b049ffa435bd57f90d5ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.259781 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70dffecdbde4a4eb6f6951187bfd499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.267913 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0c7f2c95095e4b52b0002c42225bf3a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.268522 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c7f2c95095e4b52b0002c42225bf3a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.269387 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a23788a3c209434db9f0ef44e0220672" } new_table_name: "default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b" modify_external_catalogs: true
I20250625 07:52:37.271864 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a6d1fa25a05b4d42aa04d2a632d23bdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.272478 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6d1fa25a05b4d42aa04d2a632d23bdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.275909 758 tablet_service.cc:1468] Processing CreateTablet for tablet 35ec542f480f4b32b59b364f3194e988 (DEFAULT_TABLE table=default.Table_6157edbcfec44ec5a0e7ccd884de1ab7 [id=794225dd6afe4705baffd9b383440638]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.276268 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9a2a353629294eab9a2a27411699f795 (DEFAULT_TABLE table=default.Table_2e441da0a6b34a83a84efaa1c2041ef2 [id=7b97175ead54406d9b940e1c29bf00f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.276391 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ec542f480f4b32b59b364f3194e988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.276654 741 tablet_service.cc:1468] Processing CreateTablet for tablet c33c1c9dc1b24e40bfb930fa47e95c97 (DEFAULT_TABLE table=default.Table_1c2cf30b339244f39bef84e1af811bdd [id=ed79b223049040d4a857184b67673975]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.276667 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2a353629294eab9a2a27411699f795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.276989 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531 (DEFAULT_TABLE table=default.Table_aaf276d4a7fd405d964f4808679f5fb1 [id=3623e3d46ee14bb99ec8deaa621cce78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.277136 748 tablet_service.cc:1468] Processing CreateTablet for tablet d8017c111cfd465290053a75955752d3 (DEFAULT_TABLE table=default.Table_3e1f71864f5a42d9a26edb7293a99df0 [id=0d93f6a3657547e9907edbf8a3160506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.277268 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.277343 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d0f1e8c3eb447c3bf8d09c69d6cc531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.277395 739 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.277613 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c33c1c9dc1b24e40bfb930fa47e95c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.277753 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.277827 752 tablet_service.cc:1468] Processing CreateTablet for tablet 35f819ef1fa74982a50d6714de305535 (DEFAULT_TABLE table=default.Table_a737026476024d37af115083c4585882 [id=b9d4f314564a43d687a82f8a5706f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.278076 755 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.278127 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35f819ef1fa74982a50d6714de305535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.278371 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.278688 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.278962 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.279271 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.279727 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.280063 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.280476 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9bcf9bdcdf294e979ace72f4fba236be (DEFAULT_TABLE table=default.Table_ff6a95b9c737428fac9ea3ef1a5e8ffd [id=bca4de7f0a564b39833ed5fc5cc88d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.281461 753 tablet_service.cc:1468] Processing CreateTablet for tablet 34ab27fe375d447f927312b50342e916 (DEFAULT_TABLE table=default.Table_a0e57062ab164088b1a7cf52ffe4dd18 [id=e1092aeef5b44b91a150aaa1413fbf4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.283246 1160 tablet_replica.cc:331] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:37.283586 1160 raft_consensus.cc:2241] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.283861 1160 raft_consensus.cc:2270] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.284798 1160 ts_tablet_manager.cc:1905] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.303556 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 76ed33a26e764dba80b601edf5454600
I20250625 07:52:37.304121 597 master_service.cc:948] ReplaceTablet: received request to replace tablet 9f30a8b59beb4311aca534ed322aff03 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:37.304077 1161 ts_tablet_manager.cc:1918] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.304334 1161 log.cc:1199] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9f4359fd842a4760bae75169e8914f1c
I20250625 07:52:37.307371 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0b83f1c3bc02423a9dabd58555dd997b (DEFAULT_TABLE table=default.Table_22c365283ad046a7bf006c4f05f78492 [id=3fdf91140bb74aefbbd065c0aa0bf80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.308840 893 tablet_service.cc:1468] Processing CreateTablet for tablet 932bd2139fa34fc38a7735e39d1b7421 (DEFAULT_TABLE table=default.Table_5b7b76e7ccd144bd913b3908d3aac577 [id=13691c48da484a9b91ddda236c6e8121]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.309278 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932bd2139fa34fc38a7735e39d1b7421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.311240 596 master_service.cc:948] ReplaceTablet: received request to replace tablet d8ab63bf0a0f4be29cb027505cedf835 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:37.312445 878 tablet_service.cc:1468] Processing CreateTablet for tablet a642a958550a40a48453f47d01b062a2 (DEFAULT_TABLE table=default.Table_57546ca5ffb04419aaa4086e381744e1 [id=90d5f3ef0abb4c0294ed82a7ae22c900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.314095 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 5c8b736b762f4b4c996c442970f58a31
I20250625 07:52:37.314713 589 master_service.cc:948] ReplaceTablet: received request to replace tablet a1a354a11c894b8f992f3d3cdd66cc06 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:37.315449 894 tablet_service.cc:1468] Processing CreateTablet for tablet dd748995940546a18024243e29af3391 (DEFAULT_TABLE table=default.Table_d5b8fff6c167473b8b81e9a4e15f4f32 [id=6490ea5ef3c54313b593ce4362ec3ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.316187 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd748995940546a18024243e29af3391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.316490 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 841d3a3c98af45339836d9849ff712fd
I20250625 07:52:37.317092 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a104d01e3b4115a8d68d59c5a371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.317246 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ccf4a2bf139a422182f3f71c58ee5e31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.317806 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccf4a2bf139a422182f3f71c58ee5e31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.324472 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b83f1c3bc02423a9dabd58555dd997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.333520 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6a084dfb44c7401b8958591d72e859d7 (DEFAULT_TABLE table=default.Table_8c8565ce30804a4f8148c11e83c7cce9 [id=1910c68dc2634716adc19ab233fd2802]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.334090 743 tablet_service.cc:1468] Processing CreateTablet for tablet cff61e3f37dc4c7b99a0aae816db1f44 (DEFAULT_TABLE table=default.Table_ae0c597df57144d3b0b8fad6aacfde2f [id=01a61afc29f643ebb926991bf3849bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.334210 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8017c111cfd465290053a75955752d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.232558 2407 rpcz_store.cc:276] 0625 07:52:36.137746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.140996 (+ 3250us) service_pool.cc:225] Handling call
0625 07:52:36.985866 (+844870us) ts_tablet_manager.cc:1145] Deleting tablet c263b671be384cd097041c60306034de
0625 07:52:36.985874 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.140379 (+154505us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.171455 (+ 31076us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.232356 (+ 60901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":844768}
I20250625 07:52:37.335417 891 tablet_service.cc:1468] Processing CreateTablet for tablet 998f5f465cc64052be2d32f639ab4705 (DEFAULT_TABLE table=default.Table_38d929aa141642c88be60057a30a40f8 [id=42d6f738997742218c9904891bc1f960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.335760 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998f5f465cc64052be2d32f639ab4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.339555 2447 raft_consensus.cc:491] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.340811 742 tablet_service.cc:1468] Processing CreateTablet for tablet d89fc1a546474a68951fd21e668f8d32 (DEFAULT_TABLE table=default.Table_b3a3855c16d64eeabd95e1bd8de74d2c [id=1df39597ee8945ca8171f0bf64c01960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.340853 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff61e3f37dc4c7b99a0aae816db1f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.343114 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a084dfb44c7401b8958591d72e859d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.344039 739 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.344242 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ab27fe375d447f927312b50342e916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.349548 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e07b663348454158a7398cfbc34444f4 (DEFAULT_TABLE table=default.Table_747b0d5eb8ca4f1ab30aced4c1431e9c [id=9821fb2d04fd4a3896059454afa6cc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.350636 883 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1588dea730407e8af8852423ce1980 (DEFAULT_TABLE table=default.Table_06ea495eb1d048f0a6cba1913dcc5406 [id=acd06b358f004612b860622d337c467e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.355388 758 tablet_service.cc:1468] Processing CreateTablet for tablet dd8fd11ecc8c4602a682a19ea7d46c1c (DEFAULT_TABLE table=default.Table_8d936848b9fc4ef384749a5a96ce3371 [id=cb0bc1e159de48c3b50b2cfc16e9cd47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.356786 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d89fc1a546474a68951fd21e668f8d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.350836 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a642a958550a40a48453f47d01b062a2 (DEFAULT_TABLE table=default.Table_57546ca5ffb04419aaa4086e381744e1 [id=90d5f3ef0abb4c0294ed82a7ae22c900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.352716 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b70dffecdbde4a4eb6f6951187bfd499 (DEFAULT_TABLE table=default.Table_976b8466313948adb4c34681120d4eac [id=447e050016b049ffa435bd57f90d5ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.353036 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 932bd2139fa34fc38a7735e39d1b7421 (DEFAULT_TABLE table=default.Table_5b7b76e7ccd144bd913b3908d3aac577 [id=13691c48da484a9b91ddda236c6e8121]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.353327 1026 tablet_service.cc:1468] Processing CreateTablet for tablet dd748995940546a18024243e29af3391 (DEFAULT_TABLE table=default.Table_d5b8fff6c167473b8b81e9a4e15f4f32 [id=6490ea5ef3c54313b593ce4362ec3ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.358577 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70dffecdbde4a4eb6f6951187bfd499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.358790 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932bd2139fa34fc38a7735e39d1b7421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.359597 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a642a958550a40a48453f47d01b062a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.360857 2447 raft_consensus.cc:513] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.361017 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 89fbdfddfc414e93898fe288b8fecb1e (DEFAULT_TABLE table=default.Table_47d9a728e38f49cf948b3bdf5ff09621 [id=c6395301fc0d4690b713be9f9cebb58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.361347 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b663348454158a7398cfbc34444f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.361488 741 tablet_service.cc:1468] Processing CreateTablet for tablet a642a958550a40a48453f47d01b062a2 (DEFAULT_TABLE table=default.Table_57546ca5ffb04419aaa4086e381744e1 [id=90d5f3ef0abb4c0294ed82a7ae22c900]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.361794 739 tablet_service.cc:1468] Processing CreateTablet for tablet 0b83f1c3bc02423a9dabd58555dd997b (DEFAULT_TABLE table=default.Table_22c365283ad046a7bf006c4f05f78492 [id=3fdf91140bb74aefbbd065c0aa0bf80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.361836 749 tablet_service.cc:1468] Processing CreateTablet for tablet e07b663348454158a7398cfbc34444f4 (DEFAULT_TABLE table=default.Table_747b0d5eb8ca4f1ab30aced4c1431e9c [id=9821fb2d04fd4a3896059454afa6cc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.362552 1161 ts_tablet_manager.cc:1939] T 9f4359fd842a4760bae75169e8914f1c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:37.362684 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 998f5f465cc64052be2d32f639ab4705 (DEFAULT_TABLE table=default.Table_38d929aa141642c88be60057a30a40f8 [id=42d6f738997742218c9904891bc1f960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.363261 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1212) took 1578 ms. Trace:
I20250625 07:52:37.363337 1161 rpcz_store.cc:276] 0625 07:52:35.784669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.784743 (+ 74us) service_pool.cc:225] Handling call
0625 07:52:37.029761 (+1245018us) ts_tablet_manager.cc:1145] Deleting tablet 9f4359fd842a4760bae75169e8914f1c
0625 07:52:37.029771 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.157675 (+127904us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.303750 (+146075us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.363243 (+ 59493us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19,"tablet-delete.queue_time_us":1244789}
I20250625 07:52:37.363947 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1217) took 1507 ms. Trace:
I20250625 07:52:37.364012 1161 rpcz_store.cc:276] 0625 07:52:35.856150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:35.856207 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:37.363821 (+1507614us) ts_tablet_manager.cc:1145] Deleting tablet d9b1ec100ca4488c9d7fc470246ffe47
0625 07:52:37.363832 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.363931 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503842}
I20250625 07:52:37.365005 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1588dea730407e8af8852423ce1980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.365482 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 70a104d01e3b4115a8d68d59c5a371a4 (DEFAULT_TABLE table=default.Table_6db07140365b4f2a8c6d6aaaf164be23 [id=25b23cd32a0a49e4a75708067548f761]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.365957 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998f5f465cc64052be2d32f639ab4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.366349 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fbdfddfc414e93898fe288b8fecb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.366503 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd748995940546a18024243e29af3391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.366709 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a104d01e3b4115a8d68d59c5a371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.367666 2447 leader_election.cc:290] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.368911 2447 raft_consensus.cc:491] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.369047 2447 raft_consensus.cc:513] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.369074 752 tablet_service.cc:1468] Processing CreateTablet for tablet 70a104d01e3b4115a8d68d59c5a371a4 (DEFAULT_TABLE table=default.Table_6db07140365b4f2a8c6d6aaaf164be23 [id=25b23cd32a0a49e4a75708067548f761]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.369338 740 tablet_service.cc:1468] Processing CreateTablet for tablet 89fbdfddfc414e93898fe288b8fecb1e (DEFAULT_TABLE table=default.Table_47d9a728e38f49cf948b3bdf5ff09621 [id=c6395301fc0d4690b713be9f9cebb58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.369537 744 tablet_service.cc:1468] Processing CreateTablet for tablet b70dffecdbde4a4eb6f6951187bfd499 (DEFAULT_TABLE table=default.Table_976b8466313948adb4c34681120d4eac [id=447e050016b049ffa435bd57f90d5ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.369688 2447 leader_election.cc:290] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.369709 745 tablet_service.cc:1468] Processing CreateTablet for tablet 932bd2139fa34fc38a7735e39d1b7421 (DEFAULT_TABLE table=default.Table_5b7b76e7ccd144bd913b3908d3aac577 [id=13691c48da484a9b91ddda236c6e8121]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.369859 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcf9bdcdf294e979ace72f4fba236be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.369936 1162 tablet_bootstrap.cc:654] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.375730 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a642a958550a40a48453f47d01b062a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.385393 1161 tablet_replica.cc:331] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:37.386523 880 tablet_service.cc:1468] Processing CreateTablet for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c (DEFAULT_TABLE table=default.Table_e8c4c4252edf421ba29135cd531dc8ae [id=0a5693c54b344f6ca8cf58a10e5a4d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.387213 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:37.388737 2407 tablet_replica.cc:331] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:37.390511 893 tablet_service.cc:1468] Processing CreateTablet for tablet 2e7a8bc8a62845f89b24cbae15514e4f (DEFAULT_TABLE table=default.Table_8555014fea674cb9873c91ccce23dee0 [id=1fe5823ec7f94613b637ec21eeafd945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.390993 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7a8bc8a62845f89b24cbae15514e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.397358 2503 tablet.cc:1722] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:37.398211 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 84f47caf3bae41b79b0aa1e320f15da4 (table default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]) successfully deleted
I20250625 07:52:37.403194 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.409317 2475 raft_consensus.cc:491] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.409502 2475 raft_consensus.cc:513] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.410072 2475 leader_election.cc:290] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.410820 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "065822c50586488785f7d9d53841a4dd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:37.411618 851 leader_election.cc:343] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.424155 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 76ed33a26e764dba80b601edf5454600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.424546 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.425359 890 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.425608 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.425807 890 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.426007 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a396bde4b5964697ad0090eab777ba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.426219 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 034903d699584feb91f35cbb2f62701a with delete_type TABLET_DATA_DELETED (replaced by tablet 37d228244cca4052889acdc8f1ddd440) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.426414 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 486463a8140c4e769cc79870d2acff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.426612 890 tablet_service.cc:1515] Processing DeleteTablet for tablet c858e286fbba450e81c6cc55f0f2b613 with delete_type TABLET_DATA_DELETED (replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.426810 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c8b736b762f4b4c996c442970f58a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.427016 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 841d3a3c98af45339836d9849ff712fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.424768 876 tablet_service.cc:1515] Processing DeleteTablet for tablet e10149f4ae0645ba88508ca51ae9f77a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.427364 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.428275 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba82c37fc0949afb51263013d9eb55e (DEFAULT_TABLE table=default.Table_6c61bacc767d4162844a114ff368d970 [id=6bb4d8b81faa4022be50000e8aec0862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.430979 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba82c37fc0949afb51263013d9eb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.437655 1163 raft_consensus.cc:357] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.437862 1163 raft_consensus.cc:383] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.437916 1163 raft_consensus.cc:738] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.438112 1163 consensus_queue.cc:260] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.438621 1163 ts_tablet_manager.cc:1428] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.211s user 0.002s sys 0.000s
I20250625 07:52:37.438937 1163 tablet_bootstrap.cc:492] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:37.441035 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8551b83c436f47d1ad74d6a8e74486da (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.441423 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8551b83c436f47d1ad74d6a8e74486da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.443094 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b83f1c3bc02423a9dabd58555dd997b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.443506 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e (DEFAULT_TABLE table=default.Table_aa08f96a75714b51bcde3e7b3dcf2c77 [id=34542fb54a7146ebb75ca477585892a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.444062 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8127db9a35cd46ee99a95fe27ab2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.445730 879 tablet_service.cc:1468] Processing CreateTablet for tablet bdb6ad2e21814a5a829bea1a9bede857 (DEFAULT_TABLE table=default.Table_90b0fb7fffff488ca122029f4343c687 [id=e62130ba2b874e1bbcf658bb553fcc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.446089 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb6ad2e21814a5a829bea1a9bede857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.446544 877 tablet_service.cc:1468] Processing CreateTablet for tablet 909989e2e35a4a20a31d286b53c9ce8e (DEFAULT_TABLE table=default.Table_f16cca6477ee43538ba95d01134aa5a3 [id=40cf2c3fe28149e389d553c6770bfda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.446861 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909989e2e35a4a20a31d286b53c9ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.389370 986 leader_election.cc:343] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:37.447685 1163 tablet_bootstrap.cc:654] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:37.447736 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 58390.8us
I20250625 07:52:37.417214 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1588dea730407e8af8852423ce1980 (DEFAULT_TABLE table=default.Table_06ea495eb1d048f0a6cba1913dcc5406 [id=acd06b358f004612b860622d337c467e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.444885 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 2e7a8bc8a62845f89b24cbae15514e4f (DEFAULT_TABLE table=default.Table_8555014fea674cb9873c91ccce23dee0 [id=1fe5823ec7f94613b637ec21eeafd945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.447021 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c (DEFAULT_TABLE table=default.Table_e8c4c4252edf421ba29135cd531dc8ae [id=0a5693c54b344f6ca8cf58a10e5a4d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.428006 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "065822c50586488785f7d9d53841a4dd" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:37.385967 1161 raft_consensus.cc:2241] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.450452 876 tablet_service.cc:1468] Processing CreateTablet for tablet 0169b85fdd6f45bbb801a2753d86e039 (DEFAULT_TABLE table=default.Table_ce8a2c36677445d2bab1c65eb72ffc37 [id=705869613f3c4108ae5981df4423675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.450320 1161 raft_consensus.cc:2270] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.453706 1161 ts_tablet_manager.cc:1905] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.453518 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c8b736b762f4b4c996c442970f58a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.454561 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 841d3a3c98af45339836d9849ff712fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.456348 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 8551b83c436f47d1ad74d6a8e74486da (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.458253 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0169b85fdd6f45bbb801a2753d86e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.462459 894 tablet_service.cc:1468] Processing CreateTablet for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1 (DEFAULT_TABLE table=default.Table_94e2dfea1b4a4e2aaf49eefe8dd523f9 [id=a0de9c82be1548df893fd41068170edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.482666 2475 raft_consensus.cc:491] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.482841 2475 raft_consensus.cc:513] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.483513 2475 leader_election.cc:290] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.483741 2447 raft_consensus.cc:491] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:37.483847 985 leader_election.cc:343] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a1baf94a051e485eaf4c0e074e147252
I20250625 07:52:37.483882 2447 raft_consensus.cc:513] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.484463 2447 leader_election.cc:290] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.484617 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9f4359fd842a4760bae75169e8914f1c (table default.Table_56a7bd0b475e451b8afa385ff9915ba5 [id=914805a4f6684943a2e76c948377f326]) successfully deleted
I20250625 07:52:37.483938 985 leader_election.cc:304] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:37.487644 2407 raft_consensus.cc:2241] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.488261 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e (DEFAULT_TABLE table=default.Table_aa08f96a75714b51bcde3e7b3dcf2c77 [id=34542fb54a7146ebb75ca477585892a2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.489663 850 leader_election.cc:343] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.489792 850 leader_election.cc:304] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.490170 2475 raft_consensus.cc:2747] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.491233 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44bfcc2b20ba4d0b83641988f2c0ee35" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:37.491480 2407 raft_consensus.cc:2270] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:37.491782 851 leader_election.cc:343] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.492084 2407 ts_tablet_manager.cc:1905] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.495319 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:37.495429 2161 raft_consensus.cc:491] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.495657 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 4da889781dbd41aebac86e9833cf2b7e (table default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba]). Expected version 1 got 0
I20250625 07:52:37.496008 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:37.496531 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:37.497138 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:37.497370 593 catalog_manager.cc:5582] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:37.497747 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:37.498134 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]). Expected version 2 got 0
I20250625 07:52:37.498282 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
W20250625 07:52:37.499833 985 leader_election.cc:343] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:37.500675 743 tablet_service.cc:1468] Processing CreateTablet for tablet 998f5f465cc64052be2d32f639ab4705 (DEFAULT_TABLE table=default.Table_38d929aa141642c88be60057a30a40f8 [id=42d6f738997742218c9904891bc1f960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.509529 748 tablet_service.cc:1468] Processing CreateTablet for tablet dd748995940546a18024243e29af3391 (DEFAULT_TABLE table=default.Table_d5b8fff6c167473b8b81e9a4e15f4f32 [id=6490ea5ef3c54313b593ce4362ec3ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.509673 2161 raft_consensus.cc:513] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:37.509951 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e07b663348454158a7398cfbc34444f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.509689 753 tablet_service.cc:1468] Processing CreateTablet for tablet 2e7a8bc8a62845f89b24cbae15514e4f (DEFAULT_TABLE table=default.Table_8555014fea674cb9873c91ccce23dee0 [id=1fe5823ec7f94613b637ec21eeafd945]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.511986 2475 raft_consensus.cc:491] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.512120 2475 raft_consensus.cc:513] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.512732 2475 leader_election.cc:290] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.514894 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3488f79d09054f6dbfc6b54c1dee0d6c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:37.515913 850 leader_election.cc:343] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.517701 2447 raft_consensus.cc:2747] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:37.496717 986 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:37.518694 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 76ed33a26e764dba80b601edf5454600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.520524 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba82c37fc0949afb51263013d9eb55e (DEFAULT_TABLE table=default.Table_6c61bacc767d4162844a114ff368d970 [id=6bb4d8b81faa4022be50000e8aec0862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.522189 746 tablet_service.cc:1468] Processing CreateTablet for tablet 8551b83c436f47d1ad74d6a8e74486da (DEFAULT_TABLE table=default.Table_d450a894040241c7bc3d4d34bca6a755 [id=461d7e07f36d4b50bfdbe179844968a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.522516 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3488f79d09054f6dbfc6b54c1dee0d6c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:37.522784 754 tablet_service.cc:1468] Processing CreateTablet for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c (DEFAULT_TABLE table=default.Table_e8c4c4252edf421ba29135cd531dc8ae [id=0a5693c54b344f6ca8cf58a10e5a4d65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.523051 851 leader_election.cc:343] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.523169 851 leader_election.cc:304] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.523326 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932bd2139fa34fc38a7735e39d1b7421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.523679 2475 raft_consensus.cc:2747] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.526106 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44bfcc2b20ba4d0b83641988f2c0ee35" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:37.527741 850 leader_election.cc:343] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.527860 850 leader_election.cc:304] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.528434 2475 raft_consensus.cc:2747] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.528852 2161 leader_election.cc:290] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.530560 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 841d3a3c98af45339836d9849ff712fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.530822 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89fbdfddfc414e93898fe288b8fecb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.531075 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998f5f465cc64052be2d32f639ab4705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.532579 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b70dffecdbde4a4eb6f6951187bfd499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.533187 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.533273 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 76ed33a26e764dba80b601edf5454600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.533349 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8551b83c436f47d1ad74d6a8e74486da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.533478 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7a8bc8a62845f89b24cbae15514e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.533666 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd748995940546a18024243e29af3391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.536247 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd8fd11ecc8c4602a682a19ea7d46c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.536315 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e (DEFAULT_TABLE table=default.Table_aa08f96a75714b51bcde3e7b3dcf2c77 [id=34542fb54a7146ebb75ca477585892a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.536732 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8127db9a35cd46ee99a95fe27ab2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.537870 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0169b85fdd6f45bbb801a2753d86e039 (DEFAULT_TABLE table=default.Table_ce8a2c36677445d2bab1c65eb72ffc37 [id=705869613f3c4108ae5981df4423675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.537988 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a104d01e3b4115a8d68d59c5a371a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.541309 742 tablet_service.cc:1468] Processing CreateTablet for tablet 8f1588dea730407e8af8852423ce1980 (DEFAULT_TABLE table=default.Table_06ea495eb1d048f0a6cba1913dcc5406 [id=acd06b358f004612b860622d337c467e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.542712 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a642a958550a40a48453f47d01b062a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.545130 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7a8bc8a62845f89b24cbae15514e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.548909 985 leader_election.cc:304] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:37.548985 747 tablet_service.cc:1468] Processing CreateTablet for tablet 0169b85fdd6f45bbb801a2753d86e039 (DEFAULT_TABLE table=default.Table_ce8a2c36677445d2bab1c65eb72ffc37 [id=705869613f3c4108ae5981df4423675f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.549358 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0169b85fdd6f45bbb801a2753d86e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.552452 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1588dea730407e8af8852423ce1980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.556488 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 (DEFAULT_TABLE table=default.Table_859d1eab20d84c2ea2ca2af35baab1da [id=11de63f18b63482288990515060f044e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.498739 593 catalog_manager.cc:5582] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:37.558095 1011 tablet_service.cc:1468] Processing CreateTablet for tablet 909989e2e35a4a20a31d286b53c9ce8e (DEFAULT_TABLE table=default.Table_f16cca6477ee43538ba95d01134aa5a3 [id=40cf2c3fe28149e389d553c6770bfda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.558254 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c8b736b762f4b4c996c442970f58a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.558456 1013 tablet_service.cc:1468] Processing CreateTablet for tablet bdb6ad2e21814a5a829bea1a9bede857 (DEFAULT_TABLE table=default.Table_90b0fb7fffff488ca122029f4343c687 [id=e62130ba2b874e1bbcf658bb553fcc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.559126 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1 (DEFAULT_TABLE table=default.Table_94e2dfea1b4a4e2aaf49eefe8dd523f9 [id=a0de9c82be1548df893fd41068170edd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.559412 985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 49583.3us
I20250625 07:52:37.559712 2447 raft_consensus.cc:2747] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.559847 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba82c37fc0949afb51263013d9eb55e (DEFAULT_TABLE table=default.Table_6c61bacc767d4162844a114ff368d970 [id=6bb4d8b81faa4022be50000e8aec0862]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.560209 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba82c37fc0949afb51263013d9eb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.560226 1158 tablet_bootstrap.cc:492] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:37.561110 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.565560 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1588dea730407e8af8852423ce1980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.568094 757 tablet_service.cc:1468] Processing CreateTablet for tablet bdb6ad2e21814a5a829bea1a9bede857 (DEFAULT_TABLE table=default.Table_90b0fb7fffff488ca122029f4343c687 [id=e62130ba2b874e1bbcf658bb553fcc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.568171 751 tablet_service.cc:1468] Processing CreateTablet for tablet 909989e2e35a4a20a31d286b53c9ce8e (DEFAULT_TABLE table=default.Table_f16cca6477ee43538ba95d01134aa5a3 [id=40cf2c3fe28149e389d553c6770bfda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.568701 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb6ad2e21814a5a829bea1a9bede857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.568990 752 tablet_service.cc:1468] Processing CreateTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 (DEFAULT_TABLE table=default.Table_859d1eab20d84c2ea2ca2af35baab1da [id=11de63f18b63482288990515060f044e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.569376 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44906d1dd7a84ec8bcc6701bdd2a1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.569599 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909989e2e35a4a20a31d286b53c9ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.570541 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0169b85fdd6f45bbb801a2753d86e039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.574558 1158 ts_tablet_manager.cc:1397] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.372s user 0.054s sys 0.097s
I20250625 07:52:37.580951 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2a66c0553ab74579b81bc475634b399c (DEFAULT_TABLE table=default.Table_c8b9e38da784424b8fbd8b3b8649a150 [id=d8f92fcaa8c247d19271e42e2f9dae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.581760 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44906d1dd7a84ec8bcc6701bdd2a1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.582039 758 tablet_service.cc:1468] Processing CreateTablet for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1 (DEFAULT_TABLE table=default.Table_94e2dfea1b4a4e2aaf49eefe8dd523f9 [id=a0de9c82be1548df893fd41068170edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.582396 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.559087 593 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:37.582932 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c07e9ab4154b4191a1d6cb78bfdbf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.583711 985 leader_election.cc:343] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:37.590229 740 tablet_service.cc:1468] Processing CreateTablet for tablet c732d0f440ad43d993a4047fc1e37172 (DEFAULT_TABLE table=default.Table_35af5e37776542d28c4089157dfe9b57 [id=8ff0503dd3e946b88b935939bb5f3e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.590572 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2a66c0553ab74579b81bc475634b399c (DEFAULT_TABLE table=default.Table_c8b9e38da784424b8fbd8b3b8649a150 [id=d8f92fcaa8c247d19271e42e2f9dae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.591220 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a2f72b227e14e46983c906520c29fa3" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:37.591398 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c732d0f440ad43d993a4047fc1e37172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.595136 880 tablet_service.cc:1468] Processing CreateTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 (DEFAULT_TABLE table=default.Table_859d1eab20d84c2ea2ca2af35baab1da [id=11de63f18b63482288990515060f044e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.585568 1020 tablet_service.cc:1468] Processing CreateTablet for tablet c732d0f440ad43d993a4047fc1e37172 (DEFAULT_TABLE table=default.Table_35af5e37776542d28c4089157dfe9b57 [id=8ff0503dd3e946b88b935939bb5f3e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.595655 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a66c0553ab74579b81bc475634b399c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.596663 1158 raft_consensus.cc:357] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.601621 747 tablet_service.cc:1468] Processing CreateTablet for tablet 2a66c0553ab74579b81bc475634b399c (DEFAULT_TABLE table=default.Table_c8b9e38da784424b8fbd8b3b8649a150 [id=d8f92fcaa8c247d19271e42e2f9dae10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.601826 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 7904c823a223408fa3ca0a324e4a1458 (DEFAULT_TABLE table=default.Table_213cb55216834eb4a87e67aff98467f9 [id=590dd400af6d47bc8507f2ab1f813d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.601979 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a66c0553ab74579b81bc475634b399c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.602301 985 leader_election.cc:304] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:37.610287 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3cc78b37d83b422fae1ed696d45953f6
I20250625 07:52:37.611250 588 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34788 (request call id 1548) took 1077 ms. Trace:
I20250625 07:52:37.611354 588 rpcz_store.cc:276] 0625 07:52:36.534270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.006271 (+472001us) service_pool.cc:225] Handling call
0625 07:52:37.015750 (+ 9479us) spinlock_profiling.cc:243] Waited 9.23 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 00007feeb473b14d 00007feeb473b243 00007feeb13fe341 00007feeb8f7fd8d 00007feeb9109da6 00007feeaff51316 00007feeaff51045 00007feeac73ec7e 00007feeac73db5e 00007feeac743d95 00007feeac7446bf
0625 07:52:37.016001 (+ 251us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0625 07:52:37.016120 (+ 119us) catalog_manager.cc:2790] Modifying in-memory table state
0625 07:52:37.016182 (+ 62us) catalog_manager.cc:2799] Locking tablets
0625 07:52:37.188619 (+172437us) catalog_manager.cc:2813] Removing table and tablets from system table
0625 07:52:37.377828 (+189209us) catalog_manager.cc:2831] Removing table from by-name map
0625 07:52:37.610222 (+232394us) catalog_manager.cc:2847] Committing in-memory state
0625 07:52:37.610260 (+ 38us) catalog_manager.cc:2851] Aborting table tasks
0625 07:52:37.611232 (+ 972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:37.301410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:37.301594 (+ 184us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:37.301601 (+ 7us) write_op.cc:435] Acquired schema lock
0625 07:52:37.301605 (+ 4us) tablet.cc:660] Decoding operations
0625 07:52:37.336889 (+ 35284us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:37.337144 (+ 255us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:37.337152 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0625 07:52:37.339187 (+ 2035us) tablet.cc:699] Row locks acquired
0625 07:52:37.339190 (+ 3us) write_op.cc:260] PREPARE: finished
0625 07:52:37.339702 (+ 512us) write_op.cc:270] Start()
0625 07:52:37.340267 (+ 565us) write_op.cc:276] Timestamp: P: 1750837957339249 usec, L: 0
0625 07:52:37.340273 (+ 6us) op_driver.cc:348] REPLICATION: starting
0625 07:52:37.368827 (+ 28554us) log.cc:844] Serialized 725 byte log entry
0625 07:52:37.372513 (+ 3686us) op_driver.cc:464] REPLICATION: finished
0625 07:52:37.376866 (+ 4353us) write_op.cc:301] APPLY: starting
0625 07:52:37.376899 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:37.376952 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:37.376996 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:37.377076 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:37.377114 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 07:52:37.377122 (+ 8us) write_op.cc:312] APPLY: finished
0625 07:52:37.377191 (+ 69us) log.cc:844] Serialized 29 byte log entry
0625 07:52:37.377243 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:37.377261 (+ 18us) write_op.cc:454] Released schema lock
0625 07:52:37.377266 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:37.377351 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:37.377356 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":21225728,"child_traces":[["op",{"apply.queue_time_us":4302,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":2,"prepare.queue_time_us":108072,"prepare.run_cpu_time_us":1327,"prepare.run_wall_time_us":69891,"raft.queue_time_us":64,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":277,"replication_time_us":32215,"spinlock_wait_cycles":714112}]]}
I20250625 07:52:37.612852 2475 raft_consensus.cc:491] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.613008 2475 raft_consensus.cc:513] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.613624 2475 leader_election.cc:290] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:37.616086 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a1a354a11c894b8f992f3d3cdd66cc06
I20250625 07:52:37.616863 589 catalog_manager.cc:6652] ReplaceTablet: tablet a1a354a11c894b8f992f3d3cdd66cc06 deleted and replaced by tablet 52af63b3b42a44ed831f202ff2601073
I20250625 07:52:37.617403 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9f30a8b59beb4311aca534ed322aff03
I20250625 07:52:37.617942 597 catalog_manager.cc:6652] ReplaceTablet: tablet 9f30a8b59beb4311aca534ed322aff03 deleted and replaced by tablet 5598fb3fe5da4acc8468c199fc7618d6
I20250625 07:52:37.618216 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_01e01fa76e6e4c7fac6a613140fab849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.618757 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01e01fa76e6e4c7fac6a613140fab849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.614233 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:37.619959 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0139149bbf574e13be3b0411a83a0593
I20250625 07:52:37.620643 592 catalog_manager.cc:6652] ReplaceTablet: tablet 0139149bbf574e13be3b0411a83a0593 deleted and replaced by tablet 73db63a514504afb977e558ae1698b4e
W20250625 07:52:37.621229 851 leader_election.cc:343] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.622834 754 tablet_service.cc:1468] Processing CreateTablet for tablet 166593ca1c234c93a71c349c8622667d (DEFAULT_TABLE table=default.Table_2a0303308113473b8a8d3f047b48e2db [id=662dc0ecbc5a4674ade0090c386b8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.622640 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8831e56790584f5b8c77f02b3e8add0a (DEFAULT_TABLE table=default.Table_76d76b6e4e9d4ab58006f74a5aa94477 [id=13214c6cbf314e1e91d603f53a59f409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.624261 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_313d5eb00cc14cc882bb2839b715189e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:37.624575 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8831e56790584f5b8c77f02b3e8add0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.624835 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_313d5eb00cc14cc882bb2839b715189e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.626094 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3613c8cb799746c294a74ca856714124" } new_table_name: "default.Table_969c5b5b88be48d9bf2fbde2ba3a016e" modify_external_catalogs: true
I20250625 07:52:37.627354 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c688c532bd1047a2b31446259d22c52d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:37.627902 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c688c532bd1047a2b31446259d22c52d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.629062 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e438577c44d4924bbfad6fa2cc3beaa" } modify_external_catalogs: true
I20250625 07:52:37.629251 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6e438577c44d4924bbfad6fa2cc3beaa" } modify_external_catalogs: true
I20250625 07:52:37.630540 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166593ca1c234c93a71c349c8622667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.630621 745 tablet_service.cc:1468] Processing CreateTablet for tablet 7904c823a223408fa3ca0a324e4a1458 (DEFAULT_TABLE table=default.Table_213cb55216834eb4a87e67aff98467f9 [id=590dd400af6d47bc8507f2ab1f813d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.631323 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0a72e472025348da8510f66904424a2b (DEFAULT_TABLE table=default.Table_bba9d21266f54573897e4dd408928ab4 [id=33da54c0ffb04dfa94a05225f4810e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.634110 1158 raft_consensus.cc:383] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.634317 1158 raft_consensus.cc:738] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.637807 886 tablet_service.cc:1468] Processing CreateTablet for tablet c732d0f440ad43d993a4047fc1e37172 (DEFAULT_TABLE table=default.Table_35af5e37776542d28c4089157dfe9b57 [id=8ff0503dd3e946b88b935939bb5f3e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.638266 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c732d0f440ad43d993a4047fc1e37172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.638151 891 tablet_service.cc:1468] Processing CreateTablet for tablet 8831e56790584f5b8c77f02b3e8add0a (DEFAULT_TABLE table=default.Table_76d76b6e4e9d4ab58006f74a5aa94477 [id=13214c6cbf314e1e91d603f53a59f409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.641106 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8831e56790584f5b8c77f02b3e8add0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.645289 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b5ed40b96fe4203ab8f5b4ca1ae9a8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:37.645664 879 tablet_service.cc:1468] Processing CreateTablet for tablet d95a5930df78400391af4060ae4657e1 (DEFAULT_TABLE table=default.Table_65c91dd95e0d4b24854320d405dfdbfc [id=fac6922bbee44ba9ad2f6fddd6016306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.645701 882 tablet_service.cc:1468] Processing CreateTablet for tablet 166593ca1c234c93a71c349c8622667d (DEFAULT_TABLE table=default.Table_2a0303308113473b8a8d3f047b48e2db [id=662dc0ecbc5a4674ade0090c386b8b65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.645866 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b5ed40b96fe4203ab8f5b4ca1ae9a8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:37.646014 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95a5930df78400391af4060ae4657e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.646363 887 tablet_service.cc:1468] Processing CreateTablet for tablet 0a72e472025348da8510f66904424a2b (DEFAULT_TABLE table=default.Table_bba9d21266f54573897e4dd408928ab4 [id=33da54c0ffb04dfa94a05225f4810e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.646710 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a72e472025348da8510f66904424a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.625540 1017 tablet_service.cc:1468] Processing CreateTablet for tablet d95a5930df78400391af4060ae4657e1 (DEFAULT_TABLE table=default.Table_65c91dd95e0d4b24854320d405dfdbfc [id=fac6922bbee44ba9ad2f6fddd6016306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.647818 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cfbfacfe8d844a6abefd8597c4cbc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.648919 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166593ca1c234c93a71c349c8622667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.649813 893 tablet_service.cc:1468] Processing CreateTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce (DEFAULT_TABLE table=default.Table_6b33905fc1b444d6876011512196d67b [id=5213dead9dca49239aff6be330263901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.650112 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670974e43d7e4886bfe5eb3cf36ab6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.652232 886 tablet_service.cc:1468] Processing CreateTablet for tablet ae70f46c7e6845879d30b30290253e41 (DEFAULT_TABLE table=default.Table_3529a591e1ea4cc5803cc8701e60808a [id=f8b7af8b486b420a8e790172486308d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.652520 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae70f46c7e6845879d30b30290253e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.653194 889 tablet_service.cc:1468] Processing CreateTablet for tablet 9381f02bdb8d42ffa0efc3144ce79f8c (DEFAULT_TABLE table=default.Table_dec54e44daf642c295a24542167b57c3 [id=9a8adb021bf94858b26433247c553ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.653477 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9381f02bdb8d42ffa0efc3144ce79f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.654186 876 tablet_service.cc:1468] Processing CreateTablet for tablet f08cb1194a5540aeb4ec5f35089d00a7 (DEFAULT_TABLE table=default.Table_9c021cd55d3e4dd5aacc7954f40aff2e [id=d47430f3a28e4108a246b9ffa4e99d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.654486 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08cb1194a5540aeb4ec5f35089d00a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.658020 1160 ts_tablet_manager.cc:1918] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.658142 1160 log.cc:1199] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/96a1f823fd3b47c9b6d1cf5647f9a68a
I20250625 07:52:37.658459 1160 ts_tablet_manager.cc:1939] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.658998 892 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6adb3fccef41f2b13e7249afeaf060 (DEFAULT_TABLE table=default.Table_5c86dbb25a2d445aa635a0dcf9043b9a [id=aa09b5a1fd544f0fbe819448cdb6cc5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.659003 883 tablet_service.cc:1468] Processing CreateTablet for tablet 1536933d04014446ac8be0a896f0e6f3 (DEFAULT_TABLE table=default.Table_672d41abf5214dcdbd0c430131712062 [id=577ad569e06a4111989fcc57e241878c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.659231 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1257) took 1060 ms. Trace:
I20250625 07:52:37.659343 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6adb3fccef41f2b13e7249afeaf060. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.659309 1160 rpcz_store.cc:276] 0625 07:52:36.598728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.598761 (+ 33us) service_pool.cc:225] Handling call
0625 07:52:37.283136 (+684375us) ts_tablet_manager.cc:1145] Deleting tablet 96a1f823fd3b47c9b6d1cf5647f9a68a
0625 07:52:37.283174 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.411617 (+128443us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.657931 (+246314us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.659216 (+ 1285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":684232}
I20250625 07:52:37.659587 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1536933d04014446ac8be0a896f0e6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.659837 2494 raft_consensus.cc:491] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.659983 2494 raft_consensus.cc:513] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.660847 878 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7b9756d9c4df3a1d96e9553192311 (DEFAULT_TABLE table=default.Table_99d809187ffc4bc2922585193e957568 [id=d7004490e26f464d83da8cda42bf1098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.661165 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7b9756d9c4df3a1d96e9553192311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.661247 753 tablet_service.cc:1468] Processing CreateTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce (DEFAULT_TABLE table=default.Table_6b33905fc1b444d6876011512196d67b [id=5213dead9dca49239aff6be330263901]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.661697 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904c823a223408fa3ca0a324e4a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.661897 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670974e43d7e4886bfe5eb3cf36ab6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.667438 883 tablet_service.cc:1468] Processing CreateTablet for tablet 0eb31e263b104da485aa44c2348516be (DEFAULT_TABLE table=default.Table_88b19fcde2d84c5e960e47a65b073f24 [id=f0440902eacf4f02bc267e89105441bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.668586 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a72e472025348da8510f66904424a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.672207 739 tablet_service.cc:1468] Processing CreateTablet for tablet d95a5930df78400391af4060ae4657e1 (DEFAULT_TABLE table=default.Table_65c91dd95e0d4b24854320d405dfdbfc [id=fac6922bbee44ba9ad2f6fddd6016306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.676095 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 96a1f823fd3b47c9b6d1cf5647f9a68a (table default.Table_7fc614d790ab4feaa0e3167c5de7cffd [id=93204c5397784d4fa49cd180005e4d5b]) successfully deleted
I20250625 07:52:37.680760 1158 consensus_queue.cc:260] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.684608 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e263b104da485aa44c2348516be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.687078 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 57c3d0810cf44a33b2acd50707931d26 (table default.Table_e7eadb1ed2af45bf8ebc908ee6e8b073 [id=e54da0da99f744839ff8cf175e23efda], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57c3d0810cf44a33b2acd50707931d26 consensus state has no leader
I20250625 07:52:37.689846 2494 leader_election.cc:290] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.690724 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d8ab63bf0a0f4be29cb027505cedf835
I20250625 07:52:37.691218 596 catalog_manager.cc:6652] ReplaceTablet: tablet d8ab63bf0a0f4be29cb027505cedf835 deleted and replaced by tablet 565d312783f24b9b995595f413ce4697
I20250625 07:52:37.691586 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_62b2ccb944794418a0e39d5874f5ef7e" } new_table_name: "default.Table_ece4c23b114840feb8d7047e6a3c053f" modify_external_catalogs: true
I20250625 07:52:37.672725 1163 tablet_bootstrap.cc:492] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:37.695091 1163 ts_tablet_manager.cc:1397] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.256s user 0.001s sys 0.011s
I20250625 07:52:37.696873 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 8831e56790584f5b8c77f02b3e8add0a (DEFAULT_TABLE table=default.Table_76d76b6e4e9d4ab58006f74a5aa94477 [id=13214c6cbf314e1e91d603f53a59f409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.697521 1163 raft_consensus.cc:357] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.697935 1163 raft_consensus.cc:383] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.698128 1163 raft_consensus.cc:738] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.698827 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9381f02bdb8d42ffa0efc3144ce79f8c (DEFAULT_TABLE table=default.Table_dec54e44daf642c295a24542167b57c3 [id=9a8adb021bf94858b26433247c553ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.698683 1163 consensus_queue.cc:260] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.673043 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.699661 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0139149bbf574e13be3b0411a83a0593 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db63a514504afb977e558ae1698b4e) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.674355 1160 tablet_replica.cc:331] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:37.699944 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 with delete_type TABLET_DATA_DELETED (replaced by tablet 565d312783f24b9b995595f413ce4697) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.678310 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f30a8b59beb4311aca534ed322aff03 with delete_type TABLET_DATA_DELETED (replaced by tablet 5598fb3fe5da4acc8468c199fc7618d6) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.705371 2447 raft_consensus.cc:2747] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.705643 2447 raft_consensus.cc:491] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.705828 2447 raft_consensus.cc:513] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.706468 2447 leader_election.cc:290] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.706637 2447 raft_consensus.cc:491] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.706768 2447 raft_consensus.cc:513] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.707417 2447 leader_election.cc:290] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.710997 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c732d0f440ad43d993a4047fc1e37172. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.711133 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:37.711661 850 leader_election.cc:343] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.711795 850 leader_election.cc:304] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.712141 1163 ts_tablet_manager.cc:1428] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250625 07:52:37.713235 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1536933d04014446ac8be0a896f0e6f3 (DEFAULT_TABLE table=default.Table_672d41abf5214dcdbd0c430131712062 [id=577ad569e06a4111989fcc57e241878c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.713749 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44906d1dd7a84ec8bcc6701bdd2a1872. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.719072 1163 tablet_bootstrap.cc:492] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:37.719656 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6adb3fccef41f2b13e7249afeaf060 (DEFAULT_TABLE table=default.Table_5c86dbb25a2d445aa635a0dcf9043b9a [id=aa09b5a1fd544f0fbe819448cdb6cc5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.720376 1160 raft_consensus.cc:2241] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.720687 1160 raft_consensus.cc:2270] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.721349 1160 ts_tablet_manager.cc:1905] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.724709 2494 raft_consensus.cc:491] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.730295 747 tablet_service.cc:1468] Processing CreateTablet for tablet 3e39a3e74e564f388985fbfadb92418c (DEFAULT_TABLE table=default.Table_4393f5cdd7ea48e6ae82d10d76875820 [id=54fc396f91194ad3a8d3d0aef9d94160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.730801 2161 raft_consensus.cc:491] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.730940 2161 raft_consensus.cc:513] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.731527 2161 leader_election.cc:290] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.735065 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 909989e2e35a4a20a31d286b53c9ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.736506 755 tablet_service.cc:1468] Processing CreateTablet for tablet ae70f46c7e6845879d30b30290253e41 (DEFAULT_TABLE table=default.Table_3529a591e1ea4cc5803cc8701e60808a [id=f8b7af8b486b420a8e790172486308d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.738624 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 with delete_type TABLET_DATA_DELETED (replaced by tablet 52af63b3b42a44ed831f202ff2601073) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.738862 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f30a8b59beb4311aca534ed322aff03 with delete_type TABLET_DATA_DELETED (replaced by tablet 5598fb3fe5da4acc8468c199fc7618d6) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.671895 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3e39a3e74e564f388985fbfadb92418c (DEFAULT_TABLE table=default.Table_4393f5cdd7ea48e6ae82d10d76875820 [id=54fc396f91194ad3a8d3d0aef9d94160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.671895 882 tablet_service.cc:1468] Processing CreateTablet for tablet e0ecfb9338ea41e5ad740db58d0ce736 (DEFAULT_TABLE table=default.Table_2172fec1936f46758baa42c0949361a9 [id=e19df199ecac4caf9722e579d3f5b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.674268 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 with delete_type TABLET_DATA_DELETED (replaced by tablet 52af63b3b42a44ed831f202ff2601073) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.637807 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7904c823a223408fa3ca0a324e4a1458 (DEFAULT_TABLE table=default.Table_213cb55216834eb4a87e67aff98467f9 [id=590dd400af6d47bc8507f2ab1f813d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.742134 1158 ts_tablet_manager.cc:1428] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.159s user 0.001s sys 0.010s
I20250625 07:52:37.743077 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0ecfb9338ea41e5ad740db58d0ce736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.743337 2494 raft_consensus.cc:513] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.743706 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904c823a223408fa3ca0a324e4a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.743938 2494 leader_election.cc:290] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.744927 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95a5930df78400391af4060ae4657e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.745903 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6adb3fccef41f2b13e7249afeaf060. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.746510 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae70f46c7e6845879d30b30290253e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.747323 1161 ts_tablet_manager.cc:1918] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.747511 1161 log.cc:1199] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/28e490ba5d08498f921b50f057b88369
I20250625 07:52:37.748039 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdb6ad2e21814a5a829bea1a9bede857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.748284 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1536933d04014446ac8be0a896f0e6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.748801 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9381f02bdb8d42ffa0efc3144ce79f8c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:37.749039 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu pid 492: Timed out: CheckLeaks RPC to 127.0.104.61:43317 timed out after 1.000s (SENT)
I20250625 07:52:37.749233 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu with pid 492
I20250625 07:52:37.749794 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 0139149bbf574e13be3b0411a83a0593 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db63a514504afb977e558ae1698b4e) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.750011 1161 ts_tablet_manager.cc:1939] T 28e490ba5d08498f921b50f057b88369 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:37.750092 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e39a3e74e564f388985fbfadb92418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.752805 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a66c0553ab74579b81bc475634b399c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.754468 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e39a3e74e564f388985fbfadb92418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.758203 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0eb31e263b104da485aa44c2348516be (DEFAULT_TABLE table=default.Table_88b19fcde2d84c5e960e47a65b073f24 [id=f0440902eacf4f02bc267e89105441bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.758666 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e263b104da485aa44c2348516be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.762485 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba82c37fc0949afb51263013d9eb55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.764802 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.765993 1160 ts_tablet_manager.cc:1918] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.770675 749 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7b9756d9c4df3a1d96e9553192311 (DEFAULT_TABLE table=default.Table_99d809187ffc4bc2922585193e957568 [id=d7004490e26f464d83da8cda42bf1098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.771296 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7904c823a223408fa3ca0a324e4a1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.771634 1160 log.cc:1199] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/32a9f4cf6838411fab06fa6d35ae9492
I20250625 07:52:37.772415 1160 ts_tablet_manager.cc:1939] T 32a9f4cf6838411fab06fa6d35ae9492 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.772404 750 tablet_service.cc:1468] Processing CreateTablet for tablet f08cb1194a5540aeb4ec5f35089d00a7 (DEFAULT_TABLE table=default.Table_9c021cd55d3e4dd5aacc7954f40aff2e [id=d47430f3a28e4108a246b9ffa4e99d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.772804 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08cb1194a5540aeb4ec5f35089d00a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.774338 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1256) took 1175 ms. Trace:
I20250625 07:52:37.776532 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7b9756d9c4df3a1d96e9553192311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.777503 1160 rpcz_store.cc:276] 0625 07:52:36.598628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.598798 (+ 170us) service_pool.cc:225] Handling call
0625 07:52:37.674307 (+1075509us) ts_tablet_manager.cc:1145] Deleting tablet 32a9f4cf6838411fab06fa6d35ae9492
0625 07:52:37.674315 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.756207 (+ 81892us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.765909 (+ 9702us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.774311 (+ 8402us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20220,"tablet-delete.queue_time_us":1075381}
I20250625 07:52:37.777778 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8551b83c436f47d1ad74d6a8e74486da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.778681 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 32a9f4cf6838411fab06fa6d35ae9492 (table default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]) successfully deleted
I20250625 07:52:37.779261 1160 tablet_replica.cc:331] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:37.780522 2407 ts_tablet_manager.cc:1918] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.780638 2407 log.cc:1199] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/7db12504a1a54add920561dae27d2d2b
I20250625 07:52:37.780932 1160 raft_consensus.cc:2241] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.781306 1160 raft_consensus.cc:2270] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.783875 1160 ts_tablet_manager.cc:1905] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.786635 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 166593ca1c234c93a71c349c8622667d (DEFAULT_TABLE table=default.Table_2a0303308113473b8a8d3f047b48e2db [id=662dc0ecbc5a4674ade0090c386b8b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.797480 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce (DEFAULT_TABLE table=default.Table_6b33905fc1b444d6876011512196d67b [id=5213dead9dca49239aff6be330263901]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.806046 986 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: ccddd630a82341cba7a857e67a349fe0
W20250625 07:52:37.806705 986 leader_election.cc:343] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0929ed76622248d1a41f08f4258108cd
I20250625 07:52:37.806947 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8127db9a35cd46ee99a95fe27ab2454e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.810155 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8831e56790584f5b8c77f02b3e8add0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.816748 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1232) took 1520 ms. Trace:
I20250625 07:52:37.803563 1163 tablet_bootstrap.cc:654] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.830282 1158 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:37.830322 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 9381f02bdb8d42ffa0efc3144ce79f8c (DEFAULT_TABLE table=default.Table_dec54e44daf642c295a24542167b57c3 [id=9a8adb021bf94858b26433247c553ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.831319 1160 ts_tablet_manager.cc:1918] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.831969 1160 log.cc:1199] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:37.833830 1015 tablet_service.cc:1468] Processing CreateTablet for tablet ae70f46c7e6845879d30b30290253e41 (DEFAULT_TABLE table=default.Table_3529a591e1ea4cc5803cc8701e60808a [id=f8b7af8b486b420a8e790172486308d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.834088 1160 ts_tablet_manager.cc:1939] T 70094e6c7e644f62bf5ef04dcfb4c380 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.834160 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1536933d04014446ac8be0a896f0e6f3 (DEFAULT_TABLE table=default.Table_672d41abf5214dcdbd0c430131712062 [id=577ad569e06a4111989fcc57e241878c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.835388 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 0a72e472025348da8510f66904424a2b (DEFAULT_TABLE table=default.Table_bba9d21266f54573897e4dd408928ab4 [id=33da54c0ffb04dfa94a05225f4810e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.838747 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95a5930df78400391af4060ae4657e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.840869 1158 tablet_bootstrap.cc:654] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.844359 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166593ca1c234c93a71c349c8622667d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.849134 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae70f46c7e6845879d30b30290253e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.852828 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670974e43d7e4886bfe5eb3cf36ab6ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.856758 1161 rpcz_store.cc:276] 0625 07:52:36.284392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.284771 (+ 379us) service_pool.cc:225] Handling call
0625 07:52:37.385272 (+1100501us) ts_tablet_manager.cc:1145] Deleting tablet 28e490ba5d08498f921b50f057b88369
0625 07:52:37.385317 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.635470 (+250153us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.744192 (+108722us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.804391 (+ 60199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061525}
I20250625 07:52:37.858764 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a72e472025348da8510f66904424a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.862972 2407 ts_tablet_manager.cc:1939] T 7db12504a1a54add920561dae27d2d2b P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:37.864882 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9381f02bdb8d42ffa0efc3144ce79f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.872972 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1233) took 1538 ms. Trace:
I20250625 07:52:37.873127 1161 rpcz_store.cc:276] 0625 07:52:36.334380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.336492 (+ 2112us) service_pool.cc:225] Handling call
0625 07:52:37.857217 (+1520725us) ts_tablet_manager.cc:1145] Deleting tablet f1a22d57d846414a8fa5855dc542b594
0625 07:52:37.872751 (+ 15534us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.872931 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1520408}
I20250625 07:52:37.873612 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1228) took 1833 ms. Trace:
I20250625 07:52:37.873703 1161 rpcz_store.cc:276] 0625 07:52:36.039966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.391782 (+351816us) service_pool.cc:225] Handling call
0625 07:52:37.873493 (+1481711us) ts_tablet_manager.cc:1145] Deleting tablet fb2f623e8c9742c28a97ce3a4b16b002
0625 07:52:37.873505 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.873594 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1480843}
I20250625 07:52:37.874042 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1234) took 1339 ms. Trace:
I20250625 07:52:37.874127 1161 rpcz_store.cc:276] 0625 07:52:36.534897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.535088 (+ 191us) service_pool.cc:225] Handling call
0625 07:52:37.873942 (+1338854us) ts_tablet_manager.cc:1145] Deleting tablet f8f803ed26a94ee081bd9a7c7eb1d660
0625 07:52:37.873952 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.874028 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338620}
I20250625 07:52:37.874392 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1235) took 1336 ms. Trace:
I20250625 07:52:37.874467 1161 rpcz_store.cc:276] 0625 07:52:36.538411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.538473 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:37.874304 (+1335831us) ts_tablet_manager.cc:1145] Deleting tablet 113c05d81d2943e38bcd62bf83a46a2e
0625 07:52:37.874311 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.874371 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335621}
I20250625 07:52:37.874723 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1236) took 1225 ms. Trace:
I20250625 07:52:37.874790 1161 rpcz_store.cc:276] 0625 07:52:36.649726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.649793 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:37.874645 (+1224852us) ts_tablet_manager.cc:1145] Deleting tablet c82b840442ec4cb4993636183812ed22
0625 07:52:37.874652 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.874712 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224674}
I20250625 07:52:37.875005 1161 tablet_replica.cc:331] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:37.875178 1161 raft_consensus.cc:2241] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.875344 1161 raft_consensus.cc:2270] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.800817 713 leader_election.cc:304] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.801946 748 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 with delete_type TABLET_DATA_DELETED (replaced by tablet 565d312783f24b9b995595f413ce4697) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:37.817628 2512 tablet.cc:1722] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:37.819409 1162 tablet_bootstrap.cc:492] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:37.796906 758 tablet_service.cc:1468] Processing CreateTablet for tablet e0ecfb9338ea41e5ad740db58d0ce736 (DEFAULT_TABLE table=default.Table_2172fec1936f46758baa42c0949361a9 [id=e19df199ecac4caf9722e579d3f5b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:37.876783 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 76002us
I20250625 07:52:37.876801 1012 tablet_service.cc:1468] Processing CreateTablet for tablet f08cb1194a5540aeb4ec5f35089d00a7 (DEFAULT_TABLE table=default.Table_9c021cd55d3e4dd5aacc7954f40aff2e [id=d47430f3a28e4108a246b9ffa4e99d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.877102 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0ecfb9338ea41e5ad740db58d0ce736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:37.877177 713 leader_election.cc:304] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.877794 2494 raft_consensus.cc:2802] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:37.878079 2161 consensus_queue.cc:1035] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:37.878489 2161 raft_consensus.cc:2802] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:37.878553 2161 raft_consensus.cc:491] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:37.878603 2161 raft_consensus.cc:3058] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:37.876811 1162 ts_tablet_manager.cc:1397] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.640s user 0.005s sys 0.026s
I20250625 07:52:37.881877 1162 raft_consensus.cc:357] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.882016 1162 raft_consensus.cc:383] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:37.882056 1162 raft_consensus.cc:738] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:37.882208 1162 consensus_queue.cc:260] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.882609 1162 ts_tablet_manager.cc:1428] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:37.882831 1162 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:37.884222 1162 tablet_bootstrap.cc:654] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:37.888985 2161 raft_consensus.cc:513] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.889935 2161 leader_election.cc:290] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:37.898430 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:37.898808 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:37.898972 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:37.899230 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 4da889781dbd41aebac86e9833cf2b7e (table default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba]). Expected version 1 got 0
I20250625 07:52:37.899394 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]). Expected version 2 got 0
I20250625 07:52:37.902693 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 1973678168a84875b747897f08dd1bad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.904561 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.904932 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.905100 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 065822c50586488785f7d9d53841a4dd with delete_type TABLET_DATA_DELETED (replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:37.912098 1161 ts_tablet_manager.cc:1905] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.913964 2521 raft_consensus.cc:2747] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.914201 2521 raft_consensus.cc:491] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:37.914330 2521 raft_consensus.cc:513] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:37.915088 2521 leader_election.cc:290] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:37.916043 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 215999us
I20250625 07:52:37.916713 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1973678168a84875b747897f08dd1bad" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:37.917733 851 leader_election.cc:343] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.939137 985 leader_election.cc:304] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:37.939608 2507 raft_consensus.cc:2747] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.943099 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]) successfully deleted
W20250625 07:52:37.943743 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f1a22d57d846414a8fa5855dc542b594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a22d57d846414a8fa5855dc542b594 already in progress: creating tablet
W20250625 07:52:37.944429 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fb2f623e8c9742c28a97ce3a4b16b002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb2f623e8c9742c28a97ce3a4b16b002 already in progress: creating tablet
I20250625 07:52:37.949895 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0eb31e263b104da485aa44c2348516be (DEFAULT_TABLE table=default.Table_88b19fcde2d84c5e960e47a65b073f24 [id=f0440902eacf4f02bc267e89105441bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.950901 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e0ecfb9338ea41e5ad740db58d0ce736 (DEFAULT_TABLE table=default.Table_2172fec1936f46758baa42c0949361a9 [id=e19df199ecac4caf9722e579d3f5b7b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.955830 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1311) took 1817 ms. Trace:
I20250625 07:52:37.956054 2407 rpcz_store.cc:276] 0625 07:52:36.137833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.141104 (+ 3271us) service_pool.cc:225] Handling call
0625 07:52:37.334856 (+1193752us) ts_tablet_manager.cc:1145] Deleting tablet 7db12504a1a54add920561dae27d2d2b
0625 07:52:37.334864 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.777731 (+442867us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.780441 (+ 2710us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.955782 (+175341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2137,"tablet-delete.queue_time_us":1193648}
I20250625 07:52:37.956648 2407 tablet_replica.cc:331] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:37.957024 2407 raft_consensus.cc:2241] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:37.957496 2407 raft_consensus.cc:2270] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.958608 2407 ts_tablet_manager.cc:1905] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.964553 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1253) took 1366 ms. Trace:
I20250625 07:52:37.964676 1160 rpcz_store.cc:276] 0625 07:52:36.598328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.598967 (+ 639us) service_pool.cc:225] Handling call
0625 07:52:37.779197 (+1180230us) ts_tablet_manager.cc:1145] Deleting tablet 70094e6c7e644f62bf5ef04dcfb4c380
0625 07:52:37.779209 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.795698 (+ 16489us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.830771 (+ 35073us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.964528 (+133757us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41665,"tablet-delete.queue_time_us":1180079}
I20250625 07:52:37.965665 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 70094e6c7e644f62bf5ef04dcfb4c380 (table default.Table_baabb7e0fd1e496bbe725e5e00574146 [id=c55921e26548445d9872a9e958f36056]) successfully deleted
I20250625 07:52:37.966750 1160 tablet_replica.cc:331] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:37.966984 1160 raft_consensus.cc:2241] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:37.967164 1160 raft_consensus.cc:2270] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:37.967757 1160 ts_tablet_manager.cc:1905] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:37.969230 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1973678168a84875b747897f08dd1bad" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:37.970525 850 leader_election.cc:343] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:37.970660 850 leader_election.cc:304] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:37.971174 2475 raft_consensus.cc:2747] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:37.979481 1160 ts_tablet_manager.cc:1918] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:37.979825 1160 log.cc:1199] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9bccda893fd5466886a4d253172309e1
I20250625 07:52:37.980556 1160 ts_tablet_manager.cc:1939] T 9bccda893fd5466886a4d253172309e1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:37.982180 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1254) took 1383 ms. Trace:
I20250625 07:52:37.995329 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 40f7b9756d9c4df3a1d96e9553192311 (DEFAULT_TABLE table=default.Table_99d809187ffc4bc2922585193e957568 [id=d7004490e26f464d83da8cda42bf1098]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:37.998435 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3e39a3e74e564f388985fbfadb92418c (DEFAULT_TABLE table=default.Table_4393f5cdd7ea48e6ae82d10d76875820 [id=54fc396f91194ad3a8d3d0aef9d94160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.000252 1163 tablet_bootstrap.cc:492] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.000660 1163 ts_tablet_manager.cc:1397] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.288s user 0.006s sys 0.000s
I20250625 07:52:38.002429 1163 raft_consensus.cc:357] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.002936 1163 raft_consensus.cc:383] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.003139 1163 raft_consensus.cc:738] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.003696 1163 consensus_queue.cc:260] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.010500 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 5e6adb3fccef41f2b13e7249afeaf060 (DEFAULT_TABLE table=default.Table_5c86dbb25a2d445aa635a0dcf9043b9a [id=aa09b5a1fd544f0fbe819448cdb6cc5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.013890 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1536933d04014446ac8be0a896f0e6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.015198 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f7b9756d9c4df3a1d96e9553192311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.016796 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e263b104da485aa44c2348516be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:38.023293 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f8f803ed26a94ee081bd9a7c7eb1d660 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8f803ed26a94ee081bd9a7c7eb1d660 already in progress: creating tablet
I20250625 07:52:38.023416 2506 raft_consensus.cc:491] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:38.023574 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 113c05d81d2943e38bcd62bf83a46a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113c05d81d2943e38bcd62bf83a46a2e already in progress: creating tablet
I20250625 07:52:38.023550 2506 raft_consensus.cc:513] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.023736 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet c82b840442ec4cb4993636183812ed22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c82b840442ec4cb4993636183812ed22 already in progress: creating tablet
I20250625 07:52:38.024112 2506 leader_election.cc:290] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.028347 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08cb1194a5540aeb4ec5f35089d00a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.031389 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f30a8b59beb4311aca534ed322aff03 with delete_type TABLET_DATA_DELETED (replaced by tablet 5598fb3fe5da4acc8468c199fc7618d6) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.031664 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 4167282133f94bfb9870bd08fc6594b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.032413 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ab63bf0a0f4be29cb027505cedf835 with delete_type TABLET_DATA_DELETED (replaced by tablet 565d312783f24b9b995595f413ce4697) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.032866 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ccddd630a82341cba7a857e67a349fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.035133 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet a1a354a11c894b8f992f3d3cdd66cc06 with delete_type TABLET_DATA_DELETED (replaced by tablet 52af63b3b42a44ed831f202ff2601073) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:38.036909 986 leader_election.cc:343] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 019959e9abed46fbbbb4755cb01cef44
W20250625 07:52:38.037753 985 leader_election.cc:343] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:38.038141 2494 raft_consensus.cc:695] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:38.038175 985 leader_election.cc:304] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.038520 2494 consensus_queue.cc:237] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.039680 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 162134us
I20250625 07:52:38.039952 2506 raft_consensus.cc:2747] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.040000 2498 consensus_queue.cc:1035] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:52:38.040443 985 leader_election.cc:343] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0929ed76622248d1a41f08f4258108cd
I20250625 07:52:38.041023 985 leader_election.cc:304] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.042604 2506 raft_consensus.cc:2747] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:38.043025 985 leader_election.cc:343] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 019959e9abed46fbbbb4755cb01cef44
I20250625 07:52:38.043541 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c263b671be384cd097041c60306034de (table default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]) successfully deleted
I20250625 07:52:38.043571 985 leader_election.cc:304] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.045174 2506 raft_consensus.cc:2747] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.046535 1048 raft_consensus.cc:3058] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.048203 1045 raft_consensus.cc:3058] T 4a2f72b227e14e46983c906520c29fa3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.054646 1162 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.054809 1162 ts_tablet_manager.cc:1397] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.002s
I20250625 07:52:38.055557 1162 raft_consensus.cc:357] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.055723 1162 raft_consensus.cc:383] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.055771 1162 raft_consensus.cc:738] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.055985 1162 consensus_queue.cc:260] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.056710 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9a51b748961b4e90971558af21f7b836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.057368 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a51b748961b4e90971558af21f7b836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.057809 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dba8780a44864169ac9cd63ab2e89fe9 consensus state has no leader
I20250625 07:52:38.060184 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 53bf5c84b12d4bb399c16eb6e77a28ef
I20250625 07:52:38.061407 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.062774 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_97d80d02416046f1b32d44edcd8ab622"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.063298 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d80d02416046f1b32d44edcd8ab622 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.064551 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e753ff3879ec40f58838c14425717b90" } new_table_name: "default.Table_a9040aa37b69422fbf38c7059883fd02" modify_external_catalogs: true
W20250625 07:52:38.066133 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a17495ae62b340258225c78df131b9c3 consensus state has no leader
I20250625 07:52:38.066444 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4801dc1c88174eb1838ebc1ebbd93d6c" } modify_external_catalogs: true
I20250625 07:52:38.066568 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4801dc1c88174eb1838ebc1ebbd93d6c" } modify_external_catalogs: true
I20250625 07:52:38.067436 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d4ce5c2fb18e406f9bcf8759d7bb67b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.067939 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4ce5c2fb18e406f9bcf8759d7bb67b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.068921 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 0139149bbf574e13be3b0411a83a0593 with delete_type TABLET_DATA_DELETED (replaced by tablet 73db63a514504afb977e558ae1698b4e) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.069294 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cc78b37d83b422fae1ed696d45953f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:37.982501 1160 rpcz_store.cc:276] 0625 07:52:36.598421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.599165 (+ 744us) service_pool.cc:225] Handling call
0625 07:52:37.965101 (+1365936us) ts_tablet_manager.cc:1145] Deleting tablet 9bccda893fd5466886a4d253172309e1
0625 07:52:37.966692 (+ 1591us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:37.973872 (+ 7180us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.979224 (+ 5352us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:37.982125 (+ 2901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365823}
I20250625 07:52:38.071123 1160 tablet_replica.cc:331] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.072003 1160 raft_consensus.cc:2241] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.072890 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_71d91cadf45d4ae1a05a5db26871f709" } modify_external_catalogs: true
I20250625 07:52:38.073082 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_71d91cadf45d4ae1a05a5db26871f709" } modify_external_catalogs: true
I20250625 07:52:38.073637 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_38d929aa141642c88be60057a30a40f8" } new_table_name: "default.Table_24648a124b794ed4b01fff64757ec6a1" modify_external_catalogs: true
I20250625 07:52:38.075049 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c9cb04ac9658429194e1997a689e4d2b" } modify_external_catalogs: true
I20250625 07:52:38.075219 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c9cb04ac9658429194e1997a689e4d2b" } modify_external_catalogs: true
I20250625 07:52:38.078801 1160 raft_consensus.cc:2270] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.080415 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0ecfb9338ea41e5ad740db58d0ce736. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.080979 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9bccda893fd5466886a4d253172309e1 (table default.Table_3910e10f6f4b41eeabfd11ac6c4343c7 [id=c90351757d464813b5d0d67936dd69c5]) successfully deleted
I20250625 07:52:38.084844 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e6adb3fccef41f2b13e7249afeaf060. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.089037 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e39a3e74e564f388985fbfadb92418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.076094 1163 ts_tablet_manager.cc:1428] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20250625 07:52:38.094291 1163 tablet_bootstrap.cc:492] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.095723 1163 tablet_bootstrap.cc:654] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.097470 1162 ts_tablet_manager.cc:1428] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20250625 07:52:38.097750 1162 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.098265 1158 tablet_bootstrap.cc:492] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.098398 1158 ts_tablet_manager.cc:1397] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.311s user 0.017s sys 0.017s
I20250625 07:52:38.099051 1158 raft_consensus.cc:357] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.099228 1158 raft_consensus.cc:383] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.099279 1162 tablet_bootstrap.cc:654] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.099282 1158 raft_consensus.cc:738] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.099490 1158 consensus_queue.cc:260] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.101537 1158 ts_tablet_manager.cc:1428] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:38.101802 1158 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.103060 1158 tablet_bootstrap.cc:654] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.104240 2506 raft_consensus.cc:491] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.104350 2506 raft_consensus.cc:513] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.104909 2506 leader_election.cc:290] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.106364 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b" } modify_external_catalogs: true
I20250625 07:52:38.106441 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_65a20180326d44a099e0216cbff039fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.106549 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b" } modify_external_catalogs: true
I20250625 07:52:38.106679 2531 tablet.cc:1722] T f7f82e83ed924d2bbedd33d96642a10a P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 07:52:38.107025 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65a20180326d44a099e0216cbff039fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.108889 986 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: bfacdde6a8ff4b6eac40d83c1d5b21f6
W20250625 07:52:38.117990 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f54fa729d3754d7eb7e0528a733d40b6 consensus state has no leader
W20250625 07:52:38.119808 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 998f5f465cc64052be2d32f639ab4705 consensus state has no leader
I20250625 07:52:38.123029 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_48759d4fd35e40ef80e2f1970c37738a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.123229 1160 ts_tablet_manager.cc:1905] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.123553 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48759d4fd35e40ef80e2f1970c37738a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.121649 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5d758b9980b7479a9ed999415a195e30" } new_table_name: "default.Table_33112255cea94c3b82797093e0f3d47f" modify_external_catalogs: true
I20250625 07:52:38.121803 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d97f4c56eb574432af661966fcda939b
I20250625 07:52:38.126495 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1857237dcf214e1d949c26f903fc6967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.126961 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1857237dcf214e1d949c26f903fc6967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.126948 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d97f4c56eb574432af661966fcda939b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.122058 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7a9fa4b0e1474b81ae419117256e95b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.128584 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a9fa4b0e1474b81ae419117256e95b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.128818 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.122642 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet abb4d9dc2da7453d8336d5589dda3a74
I20250625 07:52:38.130343 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_3e24931f1a5c4745b7a6a7d204738c4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.130520 886 tablet_service.cc:1515] Processing DeleteTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:38.130800 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e24931f1a5c4745b7a6a7d204738c4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.136724 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a90bb52358f4c38a540ad025e9e0430" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.139007 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:38.140221 985 leader_election.cc:343] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:38.140417 985 leader_election.cc:304] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.140683 2506 raft_consensus.cc:2747] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.141114 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.142221 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:38.142504 907 raft_consensus.cc:2466] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:38.144114 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d97f4c56eb574432af661966fcda939b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.144973 1161 ts_tablet_manager.cc:1918] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.145170 1161 log.cc:1199] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/871b603cb92249c6b0f16c825bf46232
I20250625 07:52:38.145565 1161 ts_tablet_manager.cc:1939] T 871b603cb92249c6b0f16c825bf46232 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.145962 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1237) took 1451 ms. Trace:
I20250625 07:52:38.146122 1161 rpcz_store.cc:276] 0625 07:52:36.694065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.694129 (+ 64us) service_pool.cc:225] Handling call
0625 07:52:37.874966 (+1180837us) ts_tablet_manager.cc:1145] Deleting tablet 871b603cb92249c6b0f16c825bf46232
0625 07:52:37.874973 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.093254 (+218281us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.144866 (+ 51612us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.145946 (+ 1080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180639}
I20250625 07:52:38.146564 1161 tablet_replica.cc:331] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.146785 1161 raft_consensus.cc:2241] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.146994 1161 raft_consensus.cc:2270] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.147625 1161 ts_tablet_manager.cc:1905] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.152233 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09120e19ea0f4ec7b94c89d173d3609f consensus state has no leader
I20250625 07:52:38.153965 1158 tablet_bootstrap.cc:492] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.154125 1158 ts_tablet_manager.cc:1397] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.001s
I20250625 07:52:38.154812 1158 raft_consensus.cc:357] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.154991 1158 raft_consensus.cc:383] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.155050 1158 raft_consensus.cc:738] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.164561 2407 ts_tablet_manager.cc:1918] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:38.165728 2506 raft_consensus.cc:491] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.165879 2506 raft_consensus.cc:513] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.166445 2506 leader_election.cc:290] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.167407 986 leader_election.cc:343] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:38.170943 1163 tablet_bootstrap.cc:492] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.171159 1163 ts_tablet_manager.cc:1397] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20250625 07:52:38.171983 1163 raft_consensus.cc:357] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.172235 1163 raft_consensus.cc:383] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.172336 1163 raft_consensus.cc:738] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.173820 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:38.174048 907 raft_consensus.cc:2466] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:38.174292 1158 consensus_queue.cc:260] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.174887 1158 ts_tablet_manager.cc:1428] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250625 07:52:38.180315 1160 ts_tablet_manager.cc:1918] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.180634 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 871b603cb92249c6b0f16c825bf46232 (table default.Table_0cfa01917c304be98bc1f8d5fb704b47 [id=945dc6444a0e4c8baf6e3d45ec619aff]) successfully deleted
I20250625 07:52:38.182328 1158 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.185559 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c50c35cac8e94e088cdacb664b8adfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.186089 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.186519 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:38.187635 985 leader_election.cc:343] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:38.187865 985 leader_election.cc:304] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.188120 2506 raft_consensus.cc:2747] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.190085 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 4b4627f4da6b4542aff1e44157087b14 (table default.Table_d5657e03390941c28681972134ee0159 [id=618d684f882f425dab0d9a7395deb004]). Expected version 3 got 0
I20250625 07:52:38.197360 1162 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.197644 1162 ts_tablet_manager.cc:1397] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.002s
I20250625 07:52:38.198565 1162 raft_consensus.cc:357] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.198827 1162 raft_consensus.cc:383] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.198935 1162 raft_consensus.cc:738] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.199254 1162 consensus_queue.cc:260] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.199837 1162 ts_tablet_manager.cc:1428] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:38.200259 1162 tablet_bootstrap.cc:492] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.206547 1161 ts_tablet_manager.cc:1918] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.206777 1161 log.cc:1199] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e289ea5553ac4ff983871ece4fc72ee5
I20250625 07:52:38.207209 1161 ts_tablet_manager.cc:1939] T e289ea5553ac4ff983871ece4fc72ee5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.212579 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d2bb8b0a8f45079083f2b42bc25011" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.215194 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1238) took 1520 ms. Trace:
I20250625 07:52:38.215471 1161 rpcz_store.cc:276] 0625 07:52:36.694200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.694249 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:38.146501 (+1452252us) ts_tablet_manager.cc:1145] Deleting tablet e289ea5553ac4ff983871ece4fc72ee5
0625 07:52:38.146519 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.150718 (+ 4199us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.206325 (+ 55607us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.215144 (+ 8819us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4480,"tablet-delete.queue_time_us":1452083}
I20250625 07:52:38.216269 1161 tablet_replica.cc:331] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.218912 1158 tablet_bootstrap.cc:654] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.234472 2407 log.cc:1199] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/28e490ba5d08498f921b50f057b88369
I20250625 07:52:38.235148 2407 ts_tablet_manager.cc:1939] T 28e490ba5d08498f921b50f057b88369 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:38.235986 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1321) took 1953 ms. Trace:
I20250625 07:52:38.236131 2407 rpcz_store.cc:276] 0625 07:52:36.282378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.282435 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:37.956573 (+1674138us) ts_tablet_manager.cc:1145] Deleting tablet 28e490ba5d08498f921b50f057b88369
0625 07:52:37.956585 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.160819 (+204234us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.164456 (+ 3637us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.235966 (+ 71510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673480}
I20250625 07:52:38.236617 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1322) took 1901 ms. Trace:
I20250625 07:52:38.236771 2407 rpcz_store.cc:276] 0625 07:52:36.335196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.335846 (+ 650us) service_pool.cc:225] Handling call
0625 07:52:38.236491 (+1900645us) ts_tablet_manager.cc:1145] Deleting tablet f1a22d57d846414a8fa5855dc542b594
0625 07:52:38.236506 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.236602 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900334}
I20250625 07:52:38.237144 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1323) took 1703 ms. Trace:
I20250625 07:52:38.237274 2407 rpcz_store.cc:276] 0625 07:52:36.533862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.533939 (+ 77us) service_pool.cc:225] Handling call
0625 07:52:38.237042 (+1703103us) ts_tablet_manager.cc:1145] Deleting tablet f8f803ed26a94ee081bd9a7c7eb1d660
0625 07:52:38.237050 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.237130 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694062}
I20250625 07:52:38.237525 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 05991b31ebf945f29fca8c060b21e7a7
I20250625 07:52:38.237715 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1324) took 1700 ms. Trace:
I20250625 07:52:38.237828 2407 rpcz_store.cc:276] 0625 07:52:36.537754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.537814 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:38.237536 (+1699722us) ts_tablet_manager.cc:1145] Deleting tablet 113c05d81d2943e38bcd62bf83a46a2e
0625 07:52:38.237544 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.237698 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1694040}
I20250625 07:52:38.238641 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1326) took 1590 ms. Trace:
I20250625 07:52:38.238756 2407 rpcz_store.cc:276] 0625 07:52:36.648325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.648482 (+ 157us) service_pool.cc:225] Handling call
0625 07:52:38.238555 (+1590073us) ts_tablet_manager.cc:1145] Deleting tablet c82b840442ec4cb4993636183812ed22
0625 07:52:38.238563 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.238627 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1589906}
I20250625 07:52:38.239073 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1328) took 1123 ms. Trace:
I20250625 07:52:38.239174 2407 rpcz_store.cc:276] 0625 07:52:37.115485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.115548 (+ 63us) service_pool.cc:225] Handling call
0625 07:52:38.238995 (+1123447us) ts_tablet_manager.cc:1145] Deleting tablet bb03e70e9aa740f18d8cd0543f730f83
0625 07:52:38.239003 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.239062 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123195}
I20250625 07:52:38.239446 2407 tablet_replica.cc:331] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:38.240808 1161 raft_consensus.cc:2241] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.240872 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 7db12504a1a54add920561dae27d2d2b (table default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]) successfully deleted
I20250625 07:52:38.241124 1161 raft_consensus.cc:2270] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.241765 1161 ts_tablet_manager.cc:1905] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.243193 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e289ea5553ac4ff983871ece4fc72ee5 (table default.Table_aacce885498f4090b6087bc73c1ba997 [id=9ce71e95731a483cab21fde651f4b31e]) successfully deleted
I20250625 07:52:38.243219 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70094e6c7e644f62bf5ef04dcfb4c380" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:38.250784 1160 log.cc:1199] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:38.201022 2534 tablet.cc:1722] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:38.243919 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.172632 1163 consensus_queue.cc:260] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.251828 2475 raft_consensus.cc:491] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.251960 2475 raft_consensus.cc:513] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.252955 2475 leader_election.cc:290] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.254334 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:38.254657 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6c0834adf1748c6a67805d26b973171" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.255060 593 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91] alter complete (version 2)
W20250625 07:52:38.258056 593 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:33657 (request call id 228) took 1038 ms (client timeout 1000 ms). Trace:
W20250625 07:52:38.258704 850 leader_election.cc:343] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:38.258170 593 rpcz_store.cc:269] 0625 07:52:37.219882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.302452 (+ 82570us) service_pool.cc:225] Handling call
0625 07:52:38.258029 (+955577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:37.665631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:37.665740 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:37.665746 (+ 6us) write_op.cc:435] Acquired schema lock
0625 07:52:37.665749 (+ 3us) tablet.cc:660] Decoding operations
0625 07:52:37.665882 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:37.665893 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:37.665898 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0625 07:52:37.665995 (+ 97us) tablet.cc:699] Row locks acquired
0625 07:52:37.665998 (+ 3us) write_op.cc:260] PREPARE: finished
0625 07:52:37.666033 (+ 35us) write_op.cc:270] Start()
0625 07:52:37.666064 (+ 31us) write_op.cc:276] Timestamp: P: 1750837957665962 usec, L: 0
0625 07:52:37.666067 (+ 3us) op_driver.cc:348] REPLICATION: starting
0625 07:52:37.666231 (+ 164us) log.cc:844] Serialized 901 byte log entry
0625 07:52:37.891589 (+225358us) op_driver.cc:464] REPLICATION: finished
0625 07:52:37.891884 (+ 295us) write_op.cc:301] APPLY: starting
0625 07:52:37.891902 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:37.891962 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:37.892008 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:37.892075 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:37.892110 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0625 07:52:37.892118 (+ 8us) write_op.cc:312] APPLY: finished
0625 07:52:37.892193 (+ 75us) log.cc:844] Serialized 29 byte log entry
0625 07:52:37.892255 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:37.892274 (+ 19us) write_op.cc:454] Released schema lock
0625 07:52:37.892280 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:37.892346 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:37.892350 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0625 07:52:37.922156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0625 07:52:37.922278 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0625 07:52:37.922284 (+ 6us) write_op.cc:435] Acquired schema lock
0625 07:52:37.922287 (+ 3us) tablet.cc:660] Decoding operations
0625 07:52:37.922385 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0625 07:52:37.922396 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0625 07:52:37.922400 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0625 07:52:37.922471 (+ 71us) tablet.cc:699] Row locks acquired
0625 07:52:37.922473 (+ 2us) write_op.cc:260] PREPARE: finished
0625 07:52:37.922510 (+ 37us) write_op.cc:270] Start()
0625 07:52:37.922547 (+ 37us) write_op.cc:276] Timestamp: P: 1750837957922424 usec, L: 0
0625 07:52:37.922551 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:38.065044 (+142493us) log.cc:844] Serialized 356 byte log entry
0625 07:52:38.065280 (+ 236us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x616000040f54. stack: 00007feea599b336 00007feea599b0f0 00007feea49913ad 0000000000971004 0000000000970fcd 000000000097099a 00007feeb13d0762 00007feeb2777855 00007feeb2776379 00007feeb277e7a0 00007feeb277e5ad 0000000000a7573c 00007feea59e09df 00007feea59e3cab 00007feea59e3b28 0000000000a7573c
0625 07:52:38.089395 (+ 24115us) op_driver.cc:464] REPLICATION: finished
0625 07:52:38.089709 (+ 314us) write_op.cc:301] APPLY: starting
0625 07:52:38.089727 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0625 07:52:38.089786 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0625 07:52:38.089789 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0625 07:52:38.089847 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0625 07:52:38.089880 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0625 07:52:38.089887 (+ 7us) write_op.cc:312] APPLY: finished
0625 07:52:38.089970 (+ 83us) log.cc:844] Serialized 23 byte log entry
0625 07:52:38.090031 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:38.090053 (+ 22us) write_op.cc:454] Released schema lock
0625 07:52:38.090059 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0625 07:52:38.090139 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0625 07:52:38.090143 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":126,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":455,"num_ops":2,"prepare.queue_time_us":56370,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":956,"replication_time_us":225486}],["op",{"apply.queue_time_us":262,"apply.run_cpu_time_us":411,"apply.run_wall_time_us":411,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":2407,"prepare.run_wall_time_us":143202,"raft.queue_time_us":1460,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":361,"replication_time_us":166803,"spinlock_wait_cycles":327264000}]]}
I20250625 07:52:38.262580 1163 ts_tablet_manager.cc:1428] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.091s user 0.001s sys 0.000s
I20250625 07:52:38.262796 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1389e17927584ad697c9c92802a0aed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.263317 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8c064e83460b4a58b1092ef7ae226598" } new_table_name: "default.Table_0b59b5eeb547459d92f3b04987393dd5" modify_external_catalogs: true
I20250625 07:52:38.263957 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_78623c0c0cf748428ad86d31b2375690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.264437 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78623c0c0cf748428ad86d31b2375690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.263350 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1389e17927584ad697c9c92802a0aed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.262939 1163 tablet_bootstrap.cc:492] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.268765 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4709f33f040149c49bd273524c9fd4f0
I20250625 07:52:38.269345 1163 tablet_bootstrap.cc:654] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.270294 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4709f33f040149c49bd273524c9fd4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.271831 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_47ab6034ce084c5d8e41812985ab3aea" } new_table_name: "default.Table_9c9b767f4f6c4113a012b180c8e85240" modify_external_catalogs: true
I20250625 07:52:38.272063 1161 ts_tablet_manager.cc:1918] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.272226 1161 log.cc:1199] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/b2bb060cdd9a48688967989d7491cfc1
I20250625 07:52:38.272536 1161 ts_tablet_manager.cc:1939] T b2bb060cdd9a48688967989d7491cfc1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.272861 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1239) took 1459 ms. Trace:
I20250625 07:52:38.272949 1161 rpcz_store.cc:276] 0625 07:52:36.813072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.813124 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:38.216085 (+1402961us) ts_tablet_manager.cc:1145] Deleting tablet b2bb060cdd9a48688967989d7491cfc1
0625 07:52:38.216094 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.247194 (+ 31100us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.271937 (+ 24743us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.272843 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402763}
I20250625 07:52:38.273278 1161 tablet_replica.cc:331] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.273223 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f5b40bffec5a46e1bee5f24f77334c4e" } modify_external_catalogs: true
I20250625 07:52:38.273371 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f5b40bffec5a46e1bee5f24f77334c4e" } modify_external_catalogs: true
I20250625 07:52:38.273458 1161 raft_consensus.cc:2241] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.273625 1161 raft_consensus.cc:2270] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.274160 1161 ts_tablet_manager.cc:1905] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.275082 1160 ts_tablet_manager.cc:1939] T 23af43b9ceeb4f2b812ee251ea8545ce P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.276039 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet b2bb060cdd9a48688967989d7491cfc1 (table default.Table_405b7023434c463bbce1552ceb099b75 [id=333372e3ebc84f19a63faeadb266781c]) successfully deleted
I20250625 07:52:38.287649 2475 raft_consensus.cc:491] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.287808 2475 raft_consensus.cc:513] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.288374 2475 leader_election.cc:290] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.289315 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40dacaa394747aaa100a0527e14eed0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:38.289892 850 leader_election.cc:343] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:38.290313 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3a76704c36b45e2afe54376e30f46fe consensus state has no leader
I20250625 07:52:38.294307 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7e91ee1c67a4449bb67de5b4a113d8f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.294358 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e6935ea883d54dd8bc633cdee1530352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.294929 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e91ee1c67a4449bb67de5b4a113d8f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.295135 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6935ea883d54dd8bc633cdee1530352 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.301592 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5c46a9100514d15a30be2d918da556f consensus state has no leader
I20250625 07:52:38.302613 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1259) took 1703 ms. Trace:
I20250625 07:52:38.305452 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.306264 1160 rpcz_store.cc:276] 0625 07:52:36.598923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.600768 (+ 1845us) service_pool.cc:225] Handling call
0625 07:52:38.071068 (+1470300us) ts_tablet_manager.cc:1145] Deleting tablet 23af43b9ceeb4f2b812ee251ea8545ce
0625 07:52:38.071081 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.176245 (+105164us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.180206 (+ 3961us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.302583 (+122377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470158}
I20250625 07:52:38.306672 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 4709f33f040149c49bd273524c9fd4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.310230 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436 (DEFAULT_TABLE table=default.Table_7b5cc617ff6f4d078c29187d9c38846d [id=73d604a104cf46a2bd5da6e0a1ed98ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.311151 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 23af43b9ceeb4f2b812ee251ea8545ce (table default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]) successfully deleted
I20250625 07:52:38.311868 1160 tablet_replica.cc:331] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.312551 1160 raft_consensus.cc:2241] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.315116 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3f1f025775d04da09eda17af4ac4fe10
I20250625 07:52:38.321909 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.242254 1162 tablet_bootstrap.cc:654] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.249747 2494 raft_consensus.cc:491] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.253686 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6c0834adf1748c6a67805d26b973171" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:38.271024 711 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250625 07:52:38.323458 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e2cd58bccd694d1c8d767d21c95e174e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.324023 2494 raft_consensus.cc:513] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.324141 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2cd58bccd694d1c8d767d21c95e174e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.324306 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40dacaa394747aaa100a0527e14eed0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:38.325152 2494 leader_election.cc:290] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.274933 713 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.330058 713 leader_election.cc:304] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.330368 2494 raft_consensus.cc:2747] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:38.330736 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 57061.6us
W20250625 07:52:38.333408 713 consensus_peers.cc:487] T c4729a62b7434cc0bcd1eecbfe838faf P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:38.334017 713 consensus_peers.cc:487] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:38.334574 713 consensus_peers.cc:487] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:38.335602 713 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.337073 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_47d9a728e38f49cf948b3bdf5ff09621" } new_table_name: "default.Table_a26f7c9faab04542be3a55c93b434d66" modify_external_catalogs: true
I20250625 07:52:38.337301 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e13b99b01ed747aeaf4c7005e326366a" } modify_external_catalogs: true
I20250625 07:52:38.337496 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e13b99b01ed747aeaf4c7005e326366a" } modify_external_catalogs: true
I20250625 07:52:38.338447 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7a9fa4b0e1474b81ae419117256e95b8" } new_table_name: "default.Table_fb8fd6ee8d994e9cabf8d6834fc42a1a" modify_external_catalogs: true
I20250625 07:52:38.339490 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_144dd070706a4d6e8d042f25ec83125e" } modify_external_catalogs: true
I20250625 07:52:38.339628 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_144dd070706a4d6e8d042f25ec83125e" } modify_external_catalogs: true
I20250625 07:52:38.340906 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_887824db43f7441e822ae633b51194bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.341207 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
W20250625 07:52:38.341387 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_887824db43f7441e822ae633b51194bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.319484 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:38.323516 851 leader_election.cc:343] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.344964 851 leader_election.cc:304] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:38.345398 851 leader_election.cc:343] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.345494 851 leader_election.cc:304] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.345786 2510 raft_consensus.cc:2747] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.345918 2510 raft_consensus.cc:2747] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.346463 1162 tablet_bootstrap.cc:492] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.346609 1162 ts_tablet_manager.cc:1397] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.007s
I20250625 07:52:38.347437 1162 raft_consensus.cc:357] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.347601 1162 raft_consensus.cc:383] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.347663 1162 raft_consensus.cc:738] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.347839 1162 consensus_queue.cc:260] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.348254 1162 ts_tablet_manager.cc:1428] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:38.348522 1162 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.349522 2518 raft_consensus.cc:491] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.349632 2518 raft_consensus.cc:513] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.350070 1158 tablet_bootstrap.cc:492] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.350144 2518 leader_election.cc:290] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.350212 1158 ts_tablet_manager.cc:1397] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.006s
I20250625 07:52:38.350322 1162 tablet_bootstrap.cc:654] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.350356 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a40dacaa394747aaa100a0527e14eed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.351125 1158 raft_consensus.cc:357] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.351295 1158 raft_consensus.cc:383] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.351342 1158 raft_consensus.cc:738] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.351464 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436 (DEFAULT_TABLE table=default.Table_7b5cc617ff6f4d078c29187d9c38846d [id=73d604a104cf46a2bd5da6e0a1ed98ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.351533 1158 consensus_queue.cc:260] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.351825 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.352037 1158 ts_tablet_manager.cc:1428] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:38.356515 1158 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.358412 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_952886aa3dd24481ac14888ebe7a441d" } new_table_name: "default.Table_76ed190456c04c89956f625d4654f801" modify_external_catalogs: true
I20250625 07:52:38.360266 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_fc8419bb058a41ccb3c725d395cd32ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.360982 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc8419bb058a41ccb3c725d395cd32ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.363215 1160 raft_consensus.cc:2270] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.363380 2407 raft_consensus.cc:2241] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:38.363627 2407 raft_consensus.cc:2270] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.363808 1160 ts_tablet_manager.cc:1905] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.364363 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 (DEFAULT_TABLE table=default.Table_656035a4978b49bd86d2e4462a67a4dc [id=56d9d9e7e67849589f6145f72007bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.364491 2407 ts_tablet_manager.cc:1905] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.344715 1163 tablet_bootstrap.cc:492] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.365065 1163 ts_tablet_manager.cc:1397] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.002s
I20250625 07:52:38.365770 1163 raft_consensus.cc:357] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.366004 1163 raft_consensus.cc:383] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.366093 1163 raft_consensus.cc:738] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.366444 1163 consensus_queue.cc:260] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.370990 1160 ts_tablet_manager.cc:1918] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.371316 1160 log.cc:1199] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:38.372014 1160 ts_tablet_manager.cc:1939] T 2bcfe7c3c54741fbb0aca69650fc8eee P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.373554 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1258) took 1774 ms. Trace:
I20250625 07:52:38.373759 1160 rpcz_store.cc:276] 0625 07:52:36.598817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.600931 (+ 2114us) service_pool.cc:225] Handling call
0625 07:52:38.311811 (+1710880us) ts_tablet_manager.cc:1145] Deleting tablet 2bcfe7c3c54741fbb0aca69650fc8eee
0625 07:52:38.311824 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.366955 (+ 55131us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.370790 (+ 3835us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.373533 (+ 2743us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1710767}
I20250625 07:52:38.374531 1160 tablet_replica.cc:331] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.374927 1160 raft_consensus.cc:2241] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:38.375622 1160 raft_consensus.cc:2270] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.376785 1160 ts_tablet_manager.cc:1905] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.381155 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 2bcfe7c3c54741fbb0aca69650fc8eee (table default.Table_aa09e516c85c4bc0abcb6bead66d3be1 [id=0d08c43a2d7a432391933600b06ce3fe]) successfully deleted
I20250625 07:52:38.382736 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_da1b8e2e55234fae900520bfecbbebe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.383471 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da1b8e2e55234fae900520bfecbbebe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.385042 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_08987651f24242e4a8e9eb2e4223006c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.385792 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08987651f24242e4a8e9eb2e4223006c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.385919 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe5c37fc9bd49c685de8f469a7af0c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.386893 2475 raft_consensus.cc:491] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.387039 2475 raft_consensus.cc:513] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.387719 2475 leader_election.cc:290] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.388427 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:38.388607 776 raft_consensus.cc:2466] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:38.389091 851 leader_election.cc:304] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:38.389554 2475 raft_consensus.cc:2802] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:38.389634 1160 ts_tablet_manager.cc:1918] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:38.389654 2475 raft_consensus.cc:491] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:38.389848 1160 log.cc:1199] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:38.389878 2475 raft_consensus.cc:3058] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.390580 1160 ts_tablet_manager.cc:1939] T 5179a144a9a0445f933ca02e79bbc122 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.391592 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.391690 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 (DEFAULT_TABLE table=default.Table_656035a4978b49bd86d2e4462a67a4dc [id=56d9d9e7e67849589f6145f72007bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.392082 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee3563a2d994e6e88d2e0715a7b3816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.392129 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1260) took 1790 ms. Trace:
I20250625 07:52:38.392261 1160 rpcz_store.cc:276] 0625 07:52:36.601781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.603526 (+ 1745us) service_pool.cc:225] Handling call
0625 07:52:38.374417 (+1770891us) ts_tablet_manager.cc:1145] Deleting tablet 5179a144a9a0445f933ca02e79bbc122
0625 07:52:38.374457 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.383694 (+ 9237us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.389435 (+ 5741us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.392072 (+ 2637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770700}
I20250625 07:52:38.392925 1160 tablet_replica.cc:331] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.393311 1160 raft_consensus.cc:2241] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.393651 1160 raft_consensus.cc:2270] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.394663 1160 ts_tablet_manager.cc:1905] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.397599 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89fbdfddfc414e93898fe288b8fecb1e consensus state has no leader
I20250625 07:52:38.399422 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d5fe1be63e9842068b71adac8fc011d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.400061 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5fe1be63e9842068b71adac8fc011d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.400679 2475 raft_consensus.cc:513] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.401369 850 leader_election.cc:343] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.402542 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:38.402732 776 raft_consensus.cc:3058] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.404960 2475 leader_election.cc:290] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.405583 1160 ts_tablet_manager.cc:1918] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.405779 1160 log.cc:1199] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c82b840442ec4cb4993636183812ed22
I20250625 07:52:38.405915 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:38.406359 1160 ts_tablet_manager.cc:1939] T c82b840442ec4cb4993636183812ed22 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.406684 776 raft_consensus.cc:2466] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:38.407187 851 leader_election.cc:304] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
W20250625 07:52:38.407650 850 leader_election.cc:343] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.407902 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1261) took 1760 ms. Trace:
I20250625 07:52:38.408002 2475 raft_consensus.cc:2802] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:38.408288 2475 raft_consensus.cc:695] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:38.408085 1160 rpcz_store.cc:276] 0625 07:52:36.647873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.647941 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:38.392849 (+1744908us) ts_tablet_manager.cc:1145] Deleting tablet c82b840442ec4cb4993636183812ed22
0625 07:52:38.392858 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.399994 (+ 7136us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.405421 (+ 5427us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.407884 (+ 2463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1744708}
I20250625 07:52:38.408669 2475 consensus_queue.cc:237] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.408950 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1262) took 1398 ms. Trace:
I20250625 07:52:38.409104 1160 rpcz_store.cc:276] 0625 07:52:37.010241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.010313 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:38.408633 (+1398320us) ts_tablet_manager.cc:1145] Deleting tablet bb03e70e9aa740f18d8cd0543f730f83
0625 07:52:38.408644 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.408904 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398087}
I20250625 07:52:38.409703 1160 tablet_replica.cc:331] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.410116 1160 raft_consensus.cc:2241] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.410445 1160 raft_consensus.cc:2270] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.411487 1160 ts_tablet_manager.cc:1905] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.415831 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5179a144a9a0445f933ca02e79bbc122 (table default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]) successfully deleted
I20250625 07:52:38.416122 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c82b840442ec4cb4993636183812ed22 (table default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]) successfully deleted
W20250625 07:52:38.416400 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet bb03e70e9aa740f18d8cd0543f730f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb03e70e9aa740f18d8cd0543f730f83 already in progress: creating tablet
I20250625 07:52:38.417178 879 tablet_service.cc:1468] Processing CreateTablet for tablet 30d55d1236d74cbc86bea5befcb06097 (DEFAULT_TABLE table=default.Table_ce18b3b362544adab4b9dc33d37ae532 [id=c4fa559453f448d1af0f60bde07d3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.417724 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d55d1236d74cbc86bea5befcb06097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.419395 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e8fa349c737f4cebb1225b20e62e3dcb
I20250625 07:52:38.420235 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.420621 886 tablet_service.cc:1468] Processing CreateTablet for tablet c8d219b0e702482989bfd7adaccc2e2a (DEFAULT_TABLE table=default.Table_614158d8af6a454aadfae0f0af108096 [id=ab664559f129418297c7cdd0a8e64e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.421032 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d219b0e702482989bfd7adaccc2e2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:38.424093 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9e73032861a4dc28832fdcc1707bfe0 has no consensus state
I20250625 07:52:38.425249 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c8d219b0e702482989bfd7adaccc2e2a (DEFAULT_TABLE table=default.Table_614158d8af6a454aadfae0f0af108096 [id=ab664559f129418297c7cdd0a8e64e76]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:38.426463 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 55669us
I20250625 07:52:38.428201 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 30d55d1236d74cbc86bea5befcb06097 (DEFAULT_TABLE table=default.Table_ce18b3b362544adab4b9dc33d37ae532 [id=c4fa559453f448d1af0f60bde07d3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.429391 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.429606 2506 raft_consensus.cc:491] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.429667 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee3563a2d994e6e88d2e0715a7b3816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.429733 2506 raft_consensus.cc:513] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.430298 2506 leader_election.cc:290] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.430645 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d55d1236d74cbc86bea5befcb06097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.431344 1158 tablet_bootstrap.cc:654] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.436460 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d219b0e702482989bfd7adaccc2e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.438133 1161 ts_tablet_manager.cc:1918] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.438236 1161 log.cc:1199] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0929ed76622248d1a41f08f4258108cd
I20250625 07:52:38.438899 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7d06e9aae7eb408c99d97e52fc9f691c
I20250625 07:52:38.439703 1161 ts_tablet_manager.cc:1939] T 0929ed76622248d1a41f08f4258108cd P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.440550 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1240) took 1587 ms. Trace:
I20250625 07:52:38.440650 1161 rpcz_store.cc:276] 0625 07:52:36.853428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:36.853495 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:38.273231 (+1419736us) ts_tablet_manager.cc:1145] Deleting tablet 0929ed76622248d1a41f08f4258108cd
0625 07:52:38.273242 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.315017 (+ 41775us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.438034 (+123017us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.440529 (+ 2495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419516}
I20250625 07:52:38.441107 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1241) took 1420 ms. Trace:
I20250625 07:52:38.441205 1161 rpcz_store.cc:276] 0625 07:52:37.020737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.020789 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:38.440993 (+1420204us) ts_tablet_manager.cc:1145] Deleting tablet bb03e70e9aa740f18d8cd0543f730f83
0625 07:52:38.441002 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.441093 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420015}
I20250625 07:52:38.441466 1161 tablet_replica.cc:331] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.441643 1161 raft_consensus.cc:2241] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.441792 1161 raft_consensus.cc:2270] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.442207 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9b08172057b049b4b6e9f459a8300dba" } modify_external_catalogs: true
I20250625 07:52:38.442333 1161 ts_tablet_manager.cc:1905] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.442405 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9b08172057b049b4b6e9f459a8300dba" } modify_external_catalogs: true
I20250625 07:52:38.443138 879 tablet_service.cc:1468] Processing CreateTablet for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95 (DEFAULT_TABLE table=default.Table_e4cbeb746c0542c689f6853adce7a070 [id=4bebc9122fb740c9a8a86183b802fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.443554 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.443663 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.444432 883 tablet_service.cc:1468] Processing CreateTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 (DEFAULT_TABLE table=default.Table_5e0793ddef83409a952052783347dad5 [id=622a3a01013641329569dd6f6526e45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.444805 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d3bff930cf4ce7941d2a9608f889b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.446733 878 tablet_service.cc:1468] Processing CreateTablet for tablet f7cd7e4ac88b4ab4a7b0d787739016a8 (DEFAULT_TABLE table=default.Table_73c6562565d24a9abf20fbef500de416 [id=693970e5bc064e9db4b15d11e981f295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.447135 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd7e4ac88b4ab4a7b0d787739016a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:38.449481 986 leader_election.cc:343] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:38.454924 1161 ts_tablet_manager.cc:1918] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.455065 1161 log.cc:1199] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a1baf94a051e485eaf4c0e074e147252
W20250625 07:52:38.455163 713 leader_election.cc:343] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.456571 883 tablet_service.cc:1468] Processing CreateTablet for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.457103 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:38.457319 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 84679.2us
W20250625 07:52:38.458173 713 leader_election.cc:343] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.460448 1160 ts_tablet_manager.cc:1918] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:38.460611 1160 log.cc:1199] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8b3c54204d5446c688cb123edbd430c1
I20250625 07:52:38.461025 1160 ts_tablet_manager.cc:1939] T 8b3c54204d5446c688cb123edbd430c1 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.461620 1163 ts_tablet_manager.cc:1428] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.096s user 0.001s sys 0.000s
I20250625 07:52:38.461895 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1265) took 1245 ms. Trace:
I20250625 07:52:38.462317 1160 rpcz_store.cc:276] 0625 07:52:37.216110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.216825 (+ 715us) service_pool.cc:225] Handling call
0625 07:52:38.409594 (+1192769us) ts_tablet_manager.cc:1145] Deleting tablet 8b3c54204d5446c688cb123edbd430c1
0625 07:52:38.409637 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.426592 (+ 16955us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.460341 (+ 33749us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.461842 (+ 1501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171609}
I20250625 07:52:38.462769 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1294) took 1144 ms. Trace:
I20250625 07:52:38.462862 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]) successfully deleted
I20250625 07:52:38.462852 1160 rpcz_store.cc:276] 0625 07:52:37.318590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.424093 (+105503us) service_pool.cc:225] Handling call
0625 07:52:38.462617 (+1038524us) ts_tablet_manager.cc:1145] Deleting tablet 76ed33a26e764dba80b601edf5454600
0625 07:52:38.462626 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.462735 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038140}
I20250625 07:52:38.463228 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1295) took 1068 ms. Trace:
W20250625 07:52:38.463320 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 76ed33a26e764dba80b601edf5454600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76ed33a26e764dba80b601edf5454600 already in progress: creating tablet
I20250625 07:52:38.463320 1160 rpcz_store.cc:276] 0625 07:52:37.395130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.424502 (+ 29372us) service_pool.cc:225] Handling call
0625 07:52:38.463102 (+1038600us) ts_tablet_manager.cc:1145] Deleting tablet 26d2bb8b0a8f45079083f2b42bc25011
0625 07:52:38.463111 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.463194 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037800}
I20250625 07:52:38.463604 1160 tablet_replica.cc:331] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:38.463693 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 26d2bb8b0a8f45079083f2b42bc25011 because the tablet was not found. No further retry: Not found: Tablet not found: 26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:38.463822 1160 raft_consensus.cc:2241] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.464015 1160 raft_consensus.cc:2270] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:38.464766 713 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:38.465330 713 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.466228 748 tablet_service.cc:1515] Processing DeleteTablet for tablet d97f4c56eb574432af661966fcda939b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.466581 748 tablet_service.cc:1515] Processing DeleteTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.466938 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 05991b31ebf945f29fca8c060b21e7a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.467296 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.467613 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.467916 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.468240 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.468556 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.468824 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 53bf5c84b12d4bb399c16eb6e77a28ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:37 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.468909 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.469139 1161 ts_tablet_manager.cc:1939] T a1baf94a051e485eaf4c0e074e147252 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.469271 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.469607 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.469628 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1242) took 1422 ms. Trace:
I20250625 07:52:38.469810 1161 rpcz_store.cc:276] 0625 07:52:37.046870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.046940 (+ 70us) service_pool.cc:225] Handling call
0625 07:52:38.441424 (+1394484us) ts_tablet_manager.cc:1145] Deleting tablet a1baf94a051e485eaf4c0e074e147252
0625 07:52:38.441435 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.445344 (+ 3909us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.454816 (+ 9472us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.469609 (+ 14793us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394208}
I20250625 07:52:38.470861 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3a4e707895c4f80995d1f26e93afe45" } modify_external_catalogs: true
I20250625 07:52:38.471040 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c3a4e707895c4f80995d1f26e93afe45" } modify_external_catalogs: true
I20250625 07:52:38.471472 1162 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.471601 1162 ts_tablet_manager.cc:1397] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.006s
I20250625 07:52:38.472383 1162 raft_consensus.cc:357] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.472536 1162 raft_consensus.cc:383] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.472587 1162 raft_consensus.cc:738] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.472793 1162 consensus_queue.cc:260] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.473274 1162 ts_tablet_manager.cc:1428] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:38.473552 1162 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.475206 1162 tablet_bootstrap.cc:654] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.464072 1163 tablet_bootstrap.cc:492] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.458546 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.459810 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.459911 1027 tablet_service.cc:1468] Processing CreateTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 (DEFAULT_TABLE table=default.Table_5e0793ddef83409a952052783347dad5 [id=622a3a01013641329569dd6f6526e45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.462692 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f7cd7e4ac88b4ab4a7b0d787739016a8 (DEFAULT_TABLE table=default.Table_73c6562565d24a9abf20fbef500de416 [id=693970e5bc064e9db4b15d11e981f295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.470479 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.470834 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 8cb030847521438e9ff0565b72f1f6ec (DEFAULT_TABLE table=default.Table_99ec121a2e4c46baacd7c0ef2e335c23 [id=77b273acdc214599b41999fe620d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.477058 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d3bff930cf4ce7941d2a9608f889b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.477279 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb030847521438e9ff0565b72f1f6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.477478 1163 tablet_bootstrap.cc:654] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.477531 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.475306 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0929ed76622248d1a41f08f4258108cd (table default.Table_000e46983c3d4c018a661c9efae36202 [id=b205e551d5cf4cb186f5871b101b11c6]) successfully deleted
I20250625 07:52:38.464574 1160 ts_tablet_manager.cc:1905] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.478259 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet bb03e70e9aa740f18d8cd0543f730f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb03e70e9aa740f18d8cd0543f730f83 already in progress: creating tablet
I20250625 07:52:38.480878 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd7e4ac88b4ab4a7b0d787739016a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.490409 879 tablet_service.cc:1468] Processing CreateTablet for tablet 8cb030847521438e9ff0565b72f1f6ec (DEFAULT_TABLE table=default.Table_99ec121a2e4c46baacd7c0ef2e335c23 [id=77b273acdc214599b41999fe620d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.490923 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb030847521438e9ff0565b72f1f6ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:38.492982 985 leader_election.cc:343] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:38.493120 985 leader_election.cc:304] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.493353 2506 raft_consensus.cc:2747] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.498889 879 tablet_service.cc:1468] Processing CreateTablet for tablet acd7c3ddfe9d4b8dbed4ed75a604205c (DEFAULT_TABLE table=default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.500456 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6e61feb39d6145c0936cfa2316ddfdab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.501094 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e61feb39d6145c0936cfa2316ddfdab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.503356 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_027918835ac5440cb3e80ee702033d60" } new_table_name: "default.Table_97b9f4a1f2d94773a14672da556d9144" modify_external_catalogs: true
I20250625 07:52:38.504770 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1bf36482f7224ad9932b73d8ce0a5116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.505283 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bf36482f7224ad9932b73d8ce0a5116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.506076 1158 tablet_bootstrap.cc:492] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.506227 1158 ts_tablet_manager.cc:1397] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.005s
I20250625 07:52:38.506987 1158 raft_consensus.cc:357] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.507177 1158 raft_consensus.cc:383] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.507227 1158 raft_consensus.cc:738] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.507486 1161 tablet_replica.cc:331] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.507517 1158 consensus_queue.cc:260] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.507813 1161 raft_consensus.cc:2241] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.455873 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95 (DEFAULT_TABLE table=default.Table_e4cbeb746c0542c689f6853adce7a070 [id=4bebc9122fb740c9a8a86183b802fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.508049 1161 raft_consensus.cc:2270] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.508591 1161 ts_tablet_manager.cc:1905] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.510004 1158 ts_tablet_manager.cc:1428] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:38.510223 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.510762 748 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.511205 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 4709f33f040149c49bd273524c9fd4f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.511538 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f1f025775d04da09eda17af4ac4fe10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.512065 748 tablet_service.cc:1468] Processing CreateTablet for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436 (DEFAULT_TABLE table=default.Table_7b5cc617ff6f4d078c29187d9c38846d [id=73d604a104cf46a2bd5da6e0a1ed98ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.512550 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a9bd6ddd2ca44f48ac98cdc7fa28436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.512549 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.518774 1161 ts_tablet_manager.cc:1918] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.522029 758 tablet_service.cc:1468] Processing CreateTablet for tablet c8d219b0e702482989bfd7adaccc2e2a (DEFAULT_TABLE table=default.Table_614158d8af6a454aadfae0f0af108096 [id=ab664559f129418297c7cdd0a8e64e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.522169 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a1baf94a051e485eaf4c0e074e147252 (table default.Table_224bc28d99ba48a09163db6d3106dfc7 [id=b22830a467a8438badab1e6c24564410]) successfully deleted
I20250625 07:52:38.522414 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d219b0e702482989bfd7adaccc2e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.524878 1162 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.525097 1162 ts_tablet_manager.cc:1397] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.003s
I20250625 07:52:38.525945 1162 raft_consensus.cc:357] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.526204 1162 raft_consensus.cc:383] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.526315 1162 raft_consensus.cc:738] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.526568 1162 consensus_queue.cc:260] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.527400 1162 ts_tablet_manager.cc:1428] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:38.527719 1162 tablet_bootstrap.cc:492] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.527729 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd7c3ddfe9d4b8dbed4ed75a604205c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.528944 748 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 (DEFAULT_TABLE table=default.Table_656035a4978b49bd86d2e4462a67a4dc [id=56d9d9e7e67849589f6145f72007bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.529361 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee3563a2d994e6e88d2e0715a7b3816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.529538 1161 log.cc:1199] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/019959e9abed46fbbbb4755cb01cef44
I20250625 07:52:38.529925 1161 ts_tablet_manager.cc:1939] T 019959e9abed46fbbbb4755cb01cef44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.530287 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1243) took 1442 ms. Trace:
I20250625 07:52:38.529384 1162 tablet_bootstrap.cc:654] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.535212 886 tablet_service.cc:1468] Processing CreateTablet for tablet e4fbad23ebab481cb2ce4dd830e25718 (DEFAULT_TABLE table=default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378 [id=a1198e87dfe740528f893a0ac865b6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.521425 1158 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.535707 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4fbad23ebab481cb2ce4dd830e25718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.535507 1161 rpcz_store.cc:276] 0625 07:52:37.087708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.087760 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:38.507432 (+1419672us) ts_tablet_manager.cc:1145] Deleting tablet 019959e9abed46fbbbb4755cb01cef44
0625 07:52:38.507442 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.512102 (+ 4660us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.518680 (+ 6578us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.530271 (+ 11591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419435}
I20250625 07:52:38.536121 1161 tablet_replica.cc:331] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.536376 1161 raft_consensus.cc:2241] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.536607 1161 raft_consensus.cc:2270] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.537099 1158 tablet_bootstrap.cc:654] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.537279 1161 ts_tablet_manager.cc:1905] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.538573 1163 tablet_bootstrap.cc:492] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.538780 1163 ts_tablet_manager.cc:1397] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20250625 07:52:38.539484 1163 raft_consensus.cc:357] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.539741 1163 raft_consensus.cc:383] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.539860 1163 raft_consensus.cc:738] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.540153 1163 consensus_queue.cc:260] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.540724 1163 ts_tablet_manager.cc:1428] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:38.541400 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e4fbad23ebab481cb2ce4dd830e25718 (DEFAULT_TABLE table=default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378 [id=a1198e87dfe740528f893a0ac865b6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.541605 1012 tablet_service.cc:1468] Processing CreateTablet for tablet e5c0d5b5da514c57af4266524bdf4f98 (DEFAULT_TABLE table=default.Table_b8f97eac18cc433b8871e352302a53c7 [id=86723305f031474a9a717e20554baec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.545047 758 tablet_service.cc:1468] Processing CreateTablet for tablet 30d55d1236d74cbc86bea5befcb06097 (DEFAULT_TABLE table=default.Table_ce18b3b362544adab4b9dc33d37ae532 [id=c4fa559453f448d1af0f60bde07d3f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.545604 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d55d1236d74cbc86bea5befcb06097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.545861 1162 tablet_bootstrap.cc:492] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.546072 1162 ts_tablet_manager.cc:1397] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250625 07:52:38.546885 1162 raft_consensus.cc:357] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.547158 1162 raft_consensus.cc:383] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.547267 1162 raft_consensus.cc:738] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.547524 1162 consensus_queue.cc:260] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.548205 1162 ts_tablet_manager.cc:1428] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:38.548559 1162 tablet_bootstrap.cc:492] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.549494 756 tablet_service.cc:1468] Processing CreateTablet for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95 (DEFAULT_TABLE table=default.Table_e4cbeb746c0542c689f6853adce7a070 [id=4bebc9122fb740c9a8a86183b802fe10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.549593 745 tablet_service.cc:1468] Processing CreateTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 (DEFAULT_TABLE table=default.Table_5e0793ddef83409a952052783347dad5 [id=622a3a01013641329569dd6f6526e45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.549846 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee6ae1ef2ea74d158b2cebbe0eaa6b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.549921 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d06e9aae7eb408c99d97e52fc9f691c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.549964 878 tablet_service.cc:1468] Processing CreateTablet for tablet e5c0d5b5da514c57af4266524bdf4f98 (DEFAULT_TABLE table=default.Table_b8f97eac18cc433b8871e352302a53c7 [id=86723305f031474a9a717e20554baec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.550393 889 tablet_service.cc:1468] Processing CreateTablet for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2 (DEFAULT_TABLE table=default.Table_5c6ea2b73e764d6da9c7b6ed2068a56a [id=68277c3474014605b2090265c9e88778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.550401 746 tablet_service.cc:1468] Processing CreateTablet for tablet f7cd7e4ac88b4ab4a7b0d787739016a8 (DEFAULT_TABLE table=default.Table_73c6562565d24a9abf20fbef500de416 [id=693970e5bc064e9db4b15d11e981f295]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.550515 885 tablet_service.cc:1468] Processing CreateTablet for tablet 60c01f0a75674c469b41fac638799afa (DEFAULT_TABLE table=default.Table_37c181e25508461b927d103f11855cf6 [id=9d4446d2b73645a096bba792c0b23396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.550760 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 019959e9abed46fbbbb4755cb01cef44 (table default.Table_2c27be0b9107412facdd1bc4d768eebb [id=ea62e41d8d904cf5ac63b9d0a62cde09]) successfully deleted
I20250625 07:52:38.550855 742 tablet_service.cc:1468] Processing CreateTablet for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8 (DEFAULT_TABLE table=default.Table_268b3ed7299240f3abb976c1f05dfda6 [id=145818427a61414c80c46db2bd686aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.551290 739 tablet_service.cc:1468] Processing CreateTablet for tablet 8cb030847521438e9ff0565b72f1f6ec (DEFAULT_TABLE table=default.Table_99ec121a2e4c46baacd7c0ef2e335c23 [id=77b273acdc214599b41999fe620d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.551697 750 tablet_service.cc:1468] Processing CreateTablet for tablet acd7c3ddfe9d4b8dbed4ed75a604205c (DEFAULT_TABLE table=default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.552235 752 tablet_service.cc:1468] Processing CreateTablet for tablet e5c0d5b5da514c57af4266524bdf4f98 (DEFAULT_TABLE table=default.Table_b8f97eac18cc433b8871e352302a53c7 [id=86723305f031474a9a717e20554baec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.552234 754 tablet_service.cc:1468] Processing CreateTablet for tablet e4fbad23ebab481cb2ce4dd830e25718 (DEFAULT_TABLE table=default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378 [id=a1198e87dfe740528f893a0ac865b6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.552771 749 tablet_service.cc:1515] Processing DeleteTablet for tablet e8fa349c737f4cebb1225b20e62e3dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.552745 755 tablet_service.cc:1468] Processing CreateTablet for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2 (DEFAULT_TABLE table=default.Table_5c6ea2b73e764d6da9c7b6ed2068a56a [id=68277c3474014605b2090265c9e88778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.553227 749 tablet_service.cc:1468] Processing CreateTablet for tablet 60c01f0a75674c469b41fac638799afa (DEFAULT_TABLE table=default.Table_37c181e25508461b927d103f11855cf6 [id=9d4446d2b73645a096bba792c0b23396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.553795 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7cd7e4ac88b4ab4a7b0d787739016a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.553958 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c01f0a75674c469b41fac638799afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.554069 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 60c01f0a75674c469b41fac638799afa (DEFAULT_TABLE table=default.Table_37c181e25508461b927d103f11855cf6 [id=9d4446d2b73645a096bba792c0b23396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.556582 1162 tablet_bootstrap.cc:654] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.557015 1027 tablet_service.cc:1468] Processing CreateTablet for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2 (DEFAULT_TABLE table=default.Table_5c6ea2b73e764d6da9c7b6ed2068a56a [id=68277c3474014605b2090265c9e88778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.557924 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558130 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c0d5b5da514c57af4266524bdf4f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558198 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c0d5b5da514c57af4266524bdf4f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558298 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558473 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4fbad23ebab481cb2ce4dd830e25718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558650 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c01f0a75674c469b41fac638799afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558671 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d3bff930cf4ce7941d2a9608f889b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.558820 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd7c3ddfe9d4b8dbed4ed75a604205c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.559067 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.560796 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb030847521438e9ff0565b72f1f6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.561672 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4fbad23ebab481cb2ce4dd830e25718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.570240 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24] alter complete (version 2)
I20250625 07:52:38.573477 1161 ts_tablet_manager.cc:1918] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:38.574433 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 28e490ba5d08498f921b50f057b88369 (table default.Table_0ec61175463840a0a8f9a373e8f3d6fa [id=e2fb87b6822f4c9496ed3b0c5c796c1a]) successfully deleted
W20250625 07:52:38.574709 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f1a22d57d846414a8fa5855dc542b594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1a22d57d846414a8fa5855dc542b594 already in progress: creating tablet
W20250625 07:52:38.574899 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f8f803ed26a94ee081bd9a7c7eb1d660 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8f803ed26a94ee081bd9a7c7eb1d660 already in progress: creating tablet
W20250625 07:52:38.575073 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 113c05d81d2943e38bcd62bf83a46a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 113c05d81d2943e38bcd62bf83a46a2e already in progress: creating tablet
W20250625 07:52:38.575245 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c82b840442ec4cb4993636183812ed22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c82b840442ec4cb4993636183812ed22 already in progress: creating tablet
W20250625 07:52:38.575456 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet bb03e70e9aa740f18d8cd0543f730f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb03e70e9aa740f18d8cd0543f730f83 already in progress: creating tablet
I20250625 07:52:38.577383 1161 log.cc:1199] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8b3c54204d5446c688cb123edbd430c1
I20250625 07:52:38.577811 1163 tablet_bootstrap.cc:492] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.578922 1161 ts_tablet_manager.cc:1939] T 8b3c54204d5446c688cb123edbd430c1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.579362 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1246) took 1385 ms. Trace:
I20250625 07:52:38.500288 1160 ts_tablet_manager.cc:1918] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.583104 2407 ts_tablet_manager.cc:1918] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:38.583424 2407 log.cc:1199] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/8b3c54204d5446c688cb123edbd430c1
I20250625 07:52:38.583775 1162 tablet_bootstrap.cc:492] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.583969 1162 ts_tablet_manager.cc:1397] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20250625 07:52:38.584777 1162 raft_consensus.cc:357] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.585067 1162 raft_consensus.cc:383] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.585175 1162 raft_consensus.cc:738] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.585844 1162 consensus_queue.cc:260] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.586553 1162 ts_tablet_manager.cc:1428] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:38.586861 1162 tablet_bootstrap.cc:492] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.581341 1160 log.cc:1199] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/ea62dfb8f20f4caeaa79ea1e85d99930
I20250625 07:52:38.582373 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a6c0834adf1748c6a67805d26b973171 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.576148 2506 raft_consensus.cc:491] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.582377 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb5bb4503c9b4d51b2d02ba9c1fe11c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.581197 1158 tablet_bootstrap.cc:492] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.588896 2506 raft_consensus.cc:513] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.589121 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c0d5b5da514c57af4266524bdf4f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.589546 2506 leader_election.cc:290] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.590540 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:38.591275 1045 raft_consensus.cc:3058] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.592864 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c01f0a75674c469b41fac638799afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.601336 593 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 1405e55703324bc7a1e26fe2c349bda7 (table default.Table_cd7eb16c71bb49cba917660f5a1c1b71 [id=bde8f673978c457e8a36b1173b315a06]). Expected version 1 got 0
I20250625 07:52:38.601938 1161 rpcz_store.cc:276] 0625 07:52:37.194058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.196954 (+ 2896us) service_pool.cc:225] Handling call
0625 07:52:38.536049 (+1339095us) ts_tablet_manager.cc:1145] Deleting tablet 8b3c54204d5446c688cb123edbd430c1
0625 07:52:38.536062 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.540428 (+ 4366us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.573358 (+ 32930us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.579344 (+ 5986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338861}
I20250625 07:52:38.602393 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1273) took 1240 ms. Trace:
I20250625 07:52:38.602412 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_68221d261cec4a288a02a90ac62d03de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.602478 1161 rpcz_store.cc:276] 0625 07:52:37.362219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.453206 (+ 90987us) service_pool.cc:225] Handling call
0625 07:52:38.602251 (+1149045us) ts_tablet_manager.cc:1145] Deleting tablet 5c8b736b762f4b4c996c442970f58a31
0625 07:52:38.602262 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.602373 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1148109}
I20250625 07:52:38.602756 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1274) took 1234 ms. Trace:
I20250625 07:52:38.602815 1161 rpcz_store.cc:276] 0625 07:52:37.367981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.454345 (+ 86364us) service_pool.cc:225] Handling call
0625 07:52:38.602669 (+1148324us) ts_tablet_manager.cc:1145] Deleting tablet 841d3a3c98af45339836d9849ff712fd
0625 07:52:38.602676 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.602742 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147716}
I20250625 07:52:38.602864 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:38.602962 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68221d261cec4a288a02a90ac62d03de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.604656 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_08ff7050ded34087b0e7b538b53b1d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.605202 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08ff7050ded34087b0e7b538b53b1d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.606598 1162 tablet_bootstrap.cc:654] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.606794 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1272) took 1244 ms. Trace:
I20250625 07:52:38.606879 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_81dacedf9d814b60986bc865cc7cbf94" } modify_external_catalogs: true
I20250625 07:52:38.607059 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_81dacedf9d814b60986bc865cc7cbf94" } modify_external_catalogs: true
I20250625 07:52:38.607988 1161 rpcz_store.cc:276] 0625 07:52:37.362142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.489326 (+127184us) service_pool.cc:225] Handling call
0625 07:52:38.602985 (+1113659us) ts_tablet_manager.cc:1145] Deleting tablet 76ed33a26e764dba80b601edf5454600
0625 07:52:38.602993 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.606762 (+ 3769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1069578}
I20250625 07:52:38.608521 1161 tablet_replica.cc:331] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.608788 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1726e9aaa49544cf98ad7271dd414da8" } new_table_name: "default.Table_251de4bd2ceb4271916181c73f124bf7" modify_external_catalogs: true
I20250625 07:52:38.612272 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8a13381cae324e9494a2497d0190a868
I20250625 07:52:38.612952 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 6e77ff1b8ca848ff8e1b4538645b4be2 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:38.613813 1161 raft_consensus.cc:2241] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.613986 1161 raft_consensus.cc:2270] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.614539 1161 ts_tablet_manager.cc:1905] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.616073 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0e8c0ff0ae524795bc111cd8aaa59cbb" } new_table_name: "default.Table_aa3b7fe03b6e4002b9542c1d66169898" modify_external_catalogs: true
I20250625 07:52:38.617666 2518 raft_consensus.cc:491] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.617800 2518 raft_consensus.cc:513] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.618089 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea115540f06e4c1bac09bfd88eabda0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.618346 2518 leader_election.cc:290] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:38.618625 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea115540f06e4c1bac09bfd88eabda0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.619174 2498 consensus_queue.cc:1035] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:38.622344 1162 tablet_bootstrap.cc:492] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.625324 1162 ts_tablet_manager.cc:1397] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20250625 07:52:38.630867 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4627f4da6b4542aff1e44157087b14" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:38.631397 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a13381cae324e9494a2497d0190a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.632162 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1796a3305cb94ccb9a0596643a736a96" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.632545 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a13381cae324e9494a2497d0190a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.632668 985 leader_election.cc:304] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.632898 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96a1f823fd3b47c9b6d1cf5647f9a68a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.633571 1158 ts_tablet_manager.cc:1397] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.112s user 0.002s sys 0.001s
I20250625 07:52:38.634330 1158 raft_consensus.cc:357] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.634493 1158 raft_consensus.cc:383] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.634548 1158 raft_consensus.cc:738] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.634738 1158 consensus_queue.cc:260] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.635284 1160 ts_tablet_manager.cc:1939] T ea62dfb8f20f4caeaa79ea1e85d99930 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.636086 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1296) took 1240 ms. Trace:
I20250625 07:52:38.636190 1160 rpcz_store.cc:276] 0625 07:52:37.395234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.425318 (+ 30084us) service_pool.cc:225] Handling call
0625 07:52:38.463539 (+1038221us) ts_tablet_manager.cc:1145] Deleting tablet ea62dfb8f20f4caeaa79ea1e85d99930
0625 07:52:38.463546 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.497212 (+ 33666us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.500166 (+ 2954us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.636028 (+135862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037985}
I20250625 07:52:38.637217 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet ea62dfb8f20f4caeaa79ea1e85d99930 (table default.Table_d375845276c6425baf08c8e94b318afc [id=74a388127a4343ff94b26af12d3d1577]) successfully deleted
I20250625 07:52:38.637375 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1297) took 1242 ms. Trace:
I20250625 07:52:38.637478 1160 rpcz_store.cc:276] 0625 07:52:37.395298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.425578 (+ 30280us) service_pool.cc:225] Handling call
0625 07:52:38.636598 (+1211020us) ts_tablet_manager.cc:1145] Deleting tablet 9a90bb52358f4c38a540ad025e9e0430
0625 07:52:38.636609 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.637340 (+ 731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210838}
I20250625 07:52:38.637790 1160 tablet_replica.cc:331] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.637984 1160 raft_consensus.cc:2241] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.638013 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]) successfully deleted
I20250625 07:52:38.638140 1160 raft_consensus.cc:2270] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:38.638366 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 5c8b736b762f4b4c996c442970f58a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8b736b762f4b4c996c442970f58a31 already in progress: creating tablet
I20250625 07:52:38.638634 1160 ts_tablet_manager.cc:1905] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.638605 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 841d3a3c98af45339836d9849ff712fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 841d3a3c98af45339836d9849ff712fd already in progress: creating tablet
W20250625 07:52:38.638967 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 76ed33a26e764dba80b601edf5454600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76ed33a26e764dba80b601edf5454600 already in progress: creating tablet
W20250625 07:52:38.639194 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9f30a8b59beb4311aca534ed322aff03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a8b59beb4311aca534ed322aff03 already in progress: creating tablet
I20250625 07:52:38.642470 2407 ts_tablet_manager.cc:1939] T 8b3c54204d5446c688cb123edbd430c1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:38.579316 1163 tablet_bootstrap.cc:654] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.644006 2498 raft_consensus.cc:491] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.644141 2498 raft_consensus.cc:513] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.642510 713 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:38.644356 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1333) took 1423 ms. Trace:
I20250625 07:52:38.644464 713 leader_election.cc:304] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.644769 2498 leader_election.cc:290] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.644831 2494 raft_consensus.cc:2747] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.644435 2407 rpcz_store.cc:276] 0625 07:52:37.221160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.224733 (+ 3573us) service_pool.cc:225] Handling call
0625 07:52:38.239410 (+1014677us) ts_tablet_manager.cc:1145] Deleting tablet 8b3c54204d5446c688cb123edbd430c1
0625 07:52:38.239417 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.472783 (+233366us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.582851 (+110068us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.644333 (+ 61482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014453}
I20250625 07:52:38.646037 2475 raft_consensus.cc:491] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.646173 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1343) took 1369 ms. Trace:
I20250625 07:52:38.646236 2475 raft_consensus.cc:513] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.646325 2407 rpcz_store.cc:276] 0625 07:52:37.277138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.277181 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:38.645973 (+1368792us) ts_tablet_manager.cc:1145] Deleting tablet 9cc1842f257d409ba0a56873f582f023
0625 07:52:38.645989 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.646127 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368547}
I20250625 07:52:38.647015 2475 leader_election.cc:290] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.647158 2407 tablet_replica.cc:331] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:38.647576 2407 raft_consensus.cc:2241] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.647912 2407 raft_consensus.cc:2270] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.648193 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dddbc1200acf4e7fa93ef85b398d5832" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:38.643215 1162 raft_consensus.cc:357] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.648720 851 leader_election.cc:343] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.648500 1162 raft_consensus.cc:383] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.649328 1162 raft_consensus.cc:738] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.649839 1162 consensus_queue.cc:260] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.650951 1162 ts_tablet_manager.cc:1428] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250625 07:52:38.651513 1162 tablet_bootstrap.cc:492] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.653586 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dddbc1200acf4e7fa93ef85b398d5832" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:38.654183 850 leader_election.cc:343] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.654320 850 leader_election.cc:304] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.654434 1158 ts_tablet_manager.cc:1428] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.021s user 0.000s sys 0.004s
I20250625 07:52:38.654618 2510 raft_consensus.cc:2747] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.654712 1158 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.655135 1021 tablet_service.cc:1468] Processing CreateTablet for tablet acd7c3ddfe9d4b8dbed4ed75a604205c (DEFAULT_TABLE table=default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:38.656509 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd7c3ddfe9d4b8dbed4ed75a604205c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:38.657085 1160 ts_tablet_manager.cc:1918] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.657218 1160 log.cc:1199] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/fe630469fe7c4f4bac28277896eddc9c
I20250625 07:52:38.657557 1160 ts_tablet_manager.cc:1939] T fe630469fe7c4f4bac28277896eddc9c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.658905 713 leader_election.cc:304] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.659228 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1298) took 1263 ms. Trace:
I20250625 07:52:38.659330 1160 rpcz_store.cc:276] 0625 07:52:37.395356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.425778 (+ 30422us) service_pool.cc:225] Handling call
0625 07:52:38.637719 (+1211941us) ts_tablet_manager.cc:1145] Deleting tablet fe630469fe7c4f4bac28277896eddc9c
0625 07:52:38.637728 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.651577 (+ 13849us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.656931 (+ 5354us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.659184 (+ 2253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211763}
I20250625 07:52:38.659456 2494 raft_consensus.cc:2802] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:38.659530 2494 raft_consensus.cc:491] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:38.659590 2494 raft_consensus.cc:3058] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.661041 1160 tablet_replica.cc:331] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.661249 1160 raft_consensus.cc:2241] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.661417 1160 raft_consensus.cc:2270] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.661957 1160 ts_tablet_manager.cc:1905] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.666062 2494 raft_consensus.cc:513] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.667290 2494 leader_election.cc:290] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:38.668098 1160 ts_tablet_manager.cc:1918] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.668233 1160 log.cc:1199] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a396bde4b5964697ad0090eab777ba0e
I20250625 07:52:38.668574 1160 ts_tablet_manager.cc:1939] T a396bde4b5964697ad0090eab777ba0e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.673421 2407 ts_tablet_manager.cc:1905] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.677201 1158 tablet_bootstrap.cc:654] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.677444 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1299) took 1282 ms. Trace:
I20250625 07:52:38.677564 1160 rpcz_store.cc:276] 0625 07:52:37.395417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.425973 (+ 30556us) service_pool.cc:225] Handling call
0625 07:52:38.660966 (+1234993us) ts_tablet_manager.cc:1145] Deleting tablet a396bde4b5964697ad0090eab777ba0e
0625 07:52:38.660978 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.665000 (+ 4022us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.667946 (+ 2946us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.677392 (+ 9446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234802}
I20250625 07:52:38.679373 1160 tablet_replica.cc:331] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.679610 1160 raft_consensus.cc:2241] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.679800 1160 raft_consensus.cc:2270] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.680155 1162 tablet_bootstrap.cc:654] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.680377 1160 ts_tablet_manager.cc:1905] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.681403 1161 ts_tablet_manager.cc:1918] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.681578 1161 log.cc:1199] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/4167282133f94bfb9870bd08fc6594b4
I20250625 07:52:38.681948 1161 ts_tablet_manager.cc:1939] T 4167282133f94bfb9870bd08fc6594b4 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:38.682192 583 catalog_manager.cc:4895] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9a90bb52358f4c38a540ad025e9e0430 because the tablet was not found. No further retry: Not found: Tablet not found: 9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:38.688602 2534 tablet.cc:1722] T e8fe6405c0ea4d6ca0257a5b485251fa P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:38.693327 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3a91007503ad43cc8f6fae30e4206df8
I20250625 07:52:38.695071 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_672d41abf5214dcdbd0c430131712062" } modify_external_catalogs: true
I20250625 07:52:38.695250 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_672d41abf5214dcdbd0c430131712062" } modify_external_catalogs: true
W20250625 07:52:38.695780 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a17495ae62b340258225c78df131b9c3 (table default.Table_ece4c23b114840feb8d7047e6a3c053f [id=689da0fa81174232bf0e83b311d363da], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a17495ae62b340258225c78df131b9c3 consensus state has no leader
I20250625 07:52:38.696972 592 catalog_manager.cc:5582] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } }
I20250625 07:52:38.697743 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0d9011da7226470ab7282dcdbc637180" } new_table_name: "default.Table_3e4796a533d44263a8627ae547f7c8ba" modify_external_catalogs: true
I20250625 07:52:38.682727 2531 tablet.cc:1722] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:38.682936 2506 raft_consensus.cc:2747] T 4b4627f4da6b4542aff1e44157087b14 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:38.698381 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b927ec564efa4ff080d22a673abb9b57 consensus state has no leader
I20250625 07:52:38.699923 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3de45b2a2d3049d788bc30869bd75d2f" } modify_external_catalogs: true
I20250625 07:52:38.700047 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3de45b2a2d3049d788bc30869bd75d2f" } modify_external_catalogs: true
I20250625 07:52:38.699997 713 leader_election.cc:304] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.700330 2494 raft_consensus.cc:2802] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:38.700413 2494 raft_consensus.cc:491] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:38.700491 2494 raft_consensus.cc:3058] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.701718 592 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:38.702816 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet fe630469fe7c4f4bac28277896eddc9c (table default.Table_bebb528a00504a7e80d68cec36278f45 [id=75a65f2d29cb4ccdb6555b7f1d0fe16a]) successfully deleted
I20250625 07:52:38.703107 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a396bde4b5964697ad0090eab777ba0e (table default.Table_824016809113454d9d932a22a8d60512 [id=a0beb6150680402fbd0035acae23bcf0]) successfully deleted
I20250625 07:52:38.704653 2494 raft_consensus.cc:513] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.705646 2494 leader_election.cc:290] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.707021 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 82868us
I20250625 07:52:38.707518 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_413a830b8d974a5b96393e3559e2af92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.708106 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_413a830b8d974a5b96393e3559e2af92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.691803 1158 tablet_bootstrap.cc:492] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.708503 1158 ts_tablet_manager.cc:1397] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.001s
I20250625 07:52:38.709335 1158 raft_consensus.cc:357] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.709488 1158 raft_consensus.cc:383] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.709533 1158 raft_consensus.cc:738] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.709872 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7e6dadd5349f42d08e3e43a15ca0c845" } new_table_name: "default.Table_802e0f6519e044c0a5b420289568590f" modify_external_catalogs: true
I20250625 07:52:38.710728 1161 tablet_replica.cc:331] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.710917 1161 raft_consensus.cc:2241] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.711647 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.714020 1161 raft_consensus.cc:2270] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.711879 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.714666 1161 ts_tablet_manager.cc:1905] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.715142 1160 ts_tablet_manager.cc:1918] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.715288 1160 log.cc:1199] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/034903d699584feb91f35cbb2f62701a
I20250625 07:52:38.715601 1160 ts_tablet_manager.cc:1939] T 034903d699584feb91f35cbb2f62701a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.716303 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1300) took 1320 ms. Trace:
I20250625 07:52:38.716409 1160 rpcz_store.cc:276] 0625 07:52:37.395485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.426174 (+ 30689us) service_pool.cc:225] Handling call
0625 07:52:38.679298 (+1253124us) ts_tablet_manager.cc:1145] Deleting tablet 034903d699584feb91f35cbb2f62701a
0625 07:52:38.679310 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.685377 (+ 6067us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.714996 (+ 29619us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.716264 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252925}
I20250625 07:52:38.711174 1158 consensus_queue.cc:260] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.717276 1158 ts_tablet_manager.cc:1428] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250625 07:52:38.717370 1163 tablet_bootstrap.cc:492] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.717617 1158 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.718163 1163 ts_tablet_manager.cc:1397] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.140s user 0.003s sys 0.023s
I20250625 07:52:38.719157 1163 raft_consensus.cc:357] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:38.706847 713 consensus_peers.cc:487] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:38.720326 1163 raft_consensus.cc:383] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:38.720346 713 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:38.720479 713 leader_election.cc:304] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.720608 1163 raft_consensus.cc:738] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.720888 1163 consensus_queue.cc:260] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.721163 2518 raft_consensus.cc:2747] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.721379 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 034903d699584feb91f35cbb2f62701a (table default.Table_071bd7e6cf2a4822b9a762cae07ceb4e [id=17f7f02b235944ffa0fdbe5c1af3492d]) successfully deleted
I20250625 07:52:38.721817 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 4167282133f94bfb9870bd08fc6594b4 (table default.Table_a3226ecfd38140ef85bca57822e323e1 [id=c72fa130420440e69aef2fec3c39f3e0]) successfully deleted
I20250625 07:52:38.722002 1163 ts_tablet_manager.cc:1428] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:38.713771 2562 tablet.cc:1722] T e8fe6405c0ea4d6ca0257a5b485251fa P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250625 07:52:38.722244 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d8ab63bf0a0f4be29cb027505cedf835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ab63bf0a0f4be29cb027505cedf835 already in progress: creating tablet
I20250625 07:52:38.722296 1163 tablet_bootstrap.cc:492] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.723937 1163 tablet_bootstrap.cc:654] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:38.727052 713 consensus_peers.cc:487] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bccda893fd5466886a4d253172309e1. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:38.727210 1160 tablet_replica.cc:331] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.727427 1160 raft_consensus.cc:2241] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:38.727437 713 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:38.727625 1160 raft_consensus.cc:2270] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.727648 713 leader_election.cc:304] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.727957 2518 raft_consensus.cc:2747] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.728221 1160 ts_tablet_manager.cc:1905] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.735802 1163 tablet_bootstrap.cc:492] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:38.738626 713 consensus_peers.cc:487] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a9f4cf6838411fab06fa6d35ae9492. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:38.739012 1163 ts_tablet_manager.cc:1397] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250625 07:52:38.740192 1163 raft_consensus.cc:357] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.741242 1158 tablet_bootstrap.cc:654] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.741614 1163 raft_consensus.cc:383] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.741868 1163 raft_consensus.cc:738] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.742269 1163 consensus_queue.cc:260] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.743232 1163 ts_tablet_manager.cc:1428] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:38.744001 1163 tablet_bootstrap.cc:492] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.747238 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b4d308b0b16a4ab6a3a3589087c94136
I20250625 07:52:38.747601 1162 tablet_bootstrap.cc:492] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.748432 1162 ts_tablet_manager.cc:1397] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.002s
I20250625 07:52:38.748996 1163 tablet_bootstrap.cc:654] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.749234 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 1405e55703324bc7a1e26fe2c349bda7 (table default.Table_cd7eb16c71bb49cba917660f5a1c1b71 [id=bde8f673978c457e8a36b1173b315a06]). Expected version 1 got 0
I20250625 07:52:38.749480 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 4b4627f4da6b4542aff1e44157087b14 (table default.Table_d5657e03390941c28681972134ee0159 [id=618d684f882f425dab0d9a7395deb004]). Expected version 3 got 0
I20250625 07:52:38.749524 1162 raft_consensus.cc:357] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.750411 590 master_service.cc:948] ReplaceTablet: received request to replace tablet dbdef039d3f842e5957c5bab85bedd42 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:38.751086 1162 raft_consensus.cc:383] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.751645 1162 raft_consensus.cc:738] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.752035 1162 consensus_queue.cc:260] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.751005 1160 ts_tablet_manager.cc:1918] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.752744 1160 log.cc:1199] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/486463a8140c4e769cc79870d2acff78
I20250625 07:52:38.753099 1160 ts_tablet_manager.cc:1939] T 486463a8140c4e769cc79870d2acff78 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.754734 886 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.754957 2531 tablet.cc:1722] T e8fe6405c0ea4d6ca0257a5b485251fa P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:38.755234 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1301) took 1359 ms. Trace:
I20250625 07:52:38.755363 1160 rpcz_store.cc:276] 0625 07:52:37.395545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.426382 (+ 30837us) service_pool.cc:225] Handling call
0625 07:52:38.727127 (+1300745us) ts_tablet_manager.cc:1145] Deleting tablet 486463a8140c4e769cc79870d2acff78
0625 07:52:38.727140 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.737813 (+ 10673us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.750873 (+ 13060us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.755190 (+ 4317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1300559}
I20250625 07:52:38.755928 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.756148 1160 tablet_replica.cc:331] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.756369 1160 raft_consensus.cc:2241] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.756554 1160 raft_consensus.cc:2270] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.757128 1160 ts_tablet_manager.cc:1905] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.757447 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 486463a8140c4e769cc79870d2acff78 (table default.Table_5e910ce70a934be2be117a115fea3380 [id=1ecc6b62c7c140f1bff20d234b427666]) successfully deleted
W20250625 07:52:38.759418 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67ea9c5fac904443a0e6004f8bebedbd consensus state has no leader
I20250625 07:52:38.760246 1162 ts_tablet_manager.cc:1428] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250625 07:52:38.760531 1162 tablet_bootstrap.cc:492] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.762214 1162 tablet_bootstrap.cc:654] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.763254 1160 ts_tablet_manager.cc:1918] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.763386 1160 log.cc:1199] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/c858e286fbba450e81c6cc55f0f2b613
I20250625 07:52:38.763684 1160 ts_tablet_manager.cc:1939] T c858e286fbba450e81c6cc55f0f2b613 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.764413 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1302) took 1368 ms. Trace:
I20250625 07:52:38.764510 1160 rpcz_store.cc:276] 0625 07:52:37.395603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.426583 (+ 30980us) service_pool.cc:225] Handling call
0625 07:52:38.756075 (+1329492us) ts_tablet_manager.cc:1145] Deleting tablet c858e286fbba450e81c6cc55f0f2b613
0625 07:52:38.756087 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.760107 (+ 4020us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.763139 (+ 3032us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.764372 (+ 1233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329301}
W20250625 07:52:38.764645 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a80efc0779a4df6a1b65f74f2745661 consensus state has no leader
I20250625 07:52:38.765863 2407 ts_tablet_manager.cc:1918] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.766064 2407 log.cc:1199] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/da895c90d6ee40379fe8181c2dbd7a1e
I20250625 07:52:38.766477 2407 ts_tablet_manager.cc:1939] T da895c90d6ee40379fe8181c2dbd7a1e P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:38.767128 1163 tablet_bootstrap.cc:492] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.767342 1163 ts_tablet_manager.cc:1397] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20250625 07:52:38.767422 1158 tablet_bootstrap.cc:492] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.767565 1158 ts_tablet_manager.cc:1397] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20250625 07:52:38.768256 1163 raft_consensus.cc:357] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.768488 1163 raft_consensus.cc:383] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.768592 1163 raft_consensus.cc:738] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.768586 1158 raft_consensus.cc:357] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.768796 1158 raft_consensus.cc:383] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.768864 1158 raft_consensus.cc:738] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.769017 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 547b758c2e6c4558bb232f14b37aa81a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.769017 1163 consensus_queue.cc:260] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.769059 1158 consensus_queue.cc:260] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.769629 1163 ts_tablet_manager.cc:1428] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:38.769953 1163 tablet_bootstrap.cc:492] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.771039 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a338005a7b44b9b0fba7a7096cfcb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.771442 1158 ts_tablet_manager.cc:1428] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:38.771636 1163 tablet_bootstrap.cc:654] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.771694 1158 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.773013 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1303) took 1377 ms. Trace:
I20250625 07:52:38.773190 1160 rpcz_store.cc:276] 0625 07:52:37.395659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.426778 (+ 31119us) service_pool.cc:225] Handling call
0625 07:52:38.772857 (+1346079us) ts_tablet_manager.cc:1145] Deleting tablet 5c8b736b762f4b4c996c442970f58a31
0625 07:52:38.772866 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.772970 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345890}
I20250625 07:52:38.773784 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1304) took 1378 ms. Trace:
I20250625 07:52:38.773931 1160 rpcz_store.cc:276] 0625 07:52:37.395718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.426980 (+ 31262us) service_pool.cc:225] Handling call
0625 07:52:38.773654 (+1346674us) ts_tablet_manager.cc:1145] Deleting tablet 841d3a3c98af45339836d9849ff712fd
0625 07:52:38.773663 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.773745 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346479}
I20250625 07:52:38.774356 1160 tablet_replica.cc:331] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.774784 1160 raft_consensus.cc:2241] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.775270 1160 raft_consensus.cc:2270] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.775266 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4a2e696e0fa8443392bb829107de445e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.775801 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a2e696e0fa8443392bb829107de445e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.776449 1160 ts_tablet_manager.cc:1905] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.779951 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet c858e286fbba450e81c6cc55f0f2b613 (table default.Table_973a2ebd7372436296e18cdf46fc490f [id=e8e74c2ac8aa4e1e8a69ba5c2c70b0e7]) successfully deleted
W20250625 07:52:38.780385 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 5c8b736b762f4b4c996c442970f58a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8b736b762f4b4c996c442970f58a31 already in progress: creating tablet
I20250625 07:52:38.780956 1161 ts_tablet_manager.cc:1918] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.785746 1160 ts_tablet_manager.cc:1918] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.785903 1160 log.cc:1199] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e10149f4ae0645ba88508ca51ae9f77a
I20250625 07:52:38.786082 1161 log.cc:1199] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/ccddd630a82341cba7a857e67a349fe0
I20250625 07:52:38.786209 1160 ts_tablet_manager.cc:1939] T e10149f4ae0645ba88508ca51ae9f77a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.781281 1158 tablet_bootstrap.cc:654] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.787051 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1292) took 1468 ms. Trace:
I20250625 07:52:38.787184 1160 rpcz_store.cc:276] 0625 07:52:37.318437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.402614 (+ 84177us) service_pool.cc:225] Handling call
0625 07:52:38.774291 (+1371677us) ts_tablet_manager.cc:1145] Deleting tablet e10149f4ae0645ba88508ca51ae9f77a
0625 07:52:38.774299 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.782348 (+ 8049us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.785617 (+ 3269us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.786984 (+ 1367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347024}
I20250625 07:52:38.787717 1161 ts_tablet_manager.cc:1939] T ccddd630a82341cba7a857e67a349fe0 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.787915 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1342) took 1510 ms. Trace:
I20250625 07:52:38.788048 2407 rpcz_store.cc:276] 0625 07:52:37.277054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.277349 (+ 295us) service_pool.cc:225] Handling call
0625 07:52:38.646783 (+1369434us) ts_tablet_manager.cc:1145] Deleting tablet da895c90d6ee40379fe8181c2dbd7a1e
0625 07:52:38.646794 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.718585 (+ 71791us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.765739 (+ 47154us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.787880 (+ 22141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369267}
I20250625 07:52:38.788496 2407 tablet_replica.cc:331] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:38.788774 2407 raft_consensus.cc:2241] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
W20250625 07:52:38.788973 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 841d3a3c98af45339836d9849ff712fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 841d3a3c98af45339836d9849ff712fd already in progress: creating tablet
I20250625 07:52:38.789153 2407 pending_rounds.cc:70] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
W20250625 07:52:38.759656 713 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 96a1f823fd3b47c9b6d1cf5647f9a68a
I20250625 07:52:38.790648 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cdbdd1e3b734463ab452bafecb75822" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:38.790926 912 raft_consensus.cc:2466] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:38.792794 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cdbdd1e3b734463ab452bafecb75822" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.793670 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "716e0c4dab584788bbb7631b4ca934bf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.793761 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 8b3c54204d5446c688cb123edbd430c1 (table default.Table_373a265cd8044801a0fba0f411968870 [id=6afb43dd884946958e5dc7031f93b069]) successfully deleted
I20250625 07:52:38.793761 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:38.793952 912 raft_consensus.cc:3058] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:38.794011 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9cc1842f257d409ba0a56873f582f023 because the tablet was not found. No further retry: Not found: Tablet not found: 9cc1842f257d409ba0a56873f582f023
I20250625 07:52:38.794119 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3823279d7e06435484f1b82836ebcf39" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:38.795190 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e10149f4ae0645ba88508ca51ae9f77a (table default.Table_8c340466a42e47dead226f74c8176f49 [id=b086d131091348dcbd465e24e269721e]) successfully deleted
I20250625 07:52:38.796977 1160 tablet_replica.cc:331] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.797200 1160 raft_consensus.cc:2241] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.797371 1160 raft_consensus.cc:2270] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.797674 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet ccddd630a82341cba7a857e67a349fe0 (table default.Table_80cbf681213f4716b1b7fd920d71b59c [id=3a15e8259a5f4b1283f0c106d55baff1]) successfully deleted
I20250625 07:52:38.797907 1160 ts_tablet_manager.cc:1905] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.798285 1161 tablet_replica.cc:331] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.798465 1161 raft_consensus.cc:2241] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.798656 1161 raft_consensus.cc:2270] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.799192 1161 ts_tablet_manager.cc:1905] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.800146 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a1a354a11c894b8f992f3d3cdd66cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a354a11c894b8f992f3d3cdd66cc06 already in progress: creating tablet
I20250625 07:52:38.800814 2498 raft_consensus.cc:491] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.800962 2498 raft_consensus.cc:513] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.801568 2498 leader_election.cc:290] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.806329 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0139149bbf574e13be3b0411a83a0593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0139149bbf574e13be3b0411a83a0593 already in progress: creating tablet
I20250625 07:52:38.810043 912 raft_consensus.cc:2466] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:38.811022 1158 tablet_bootstrap.cc:492] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.811193 1158 ts_tablet_manager.cc:1397] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20250625 07:52:38.811967 1158 raft_consensus.cc:357] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.812175 1158 raft_consensus.cc:383] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.812234 1158 raft_consensus.cc:738] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:38.812397 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3cc78b37d83b422fae1ed696d45953f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc78b37d83b422fae1ed696d45953f6 already in progress: creating tablet
I20250625 07:52:38.812431 1160 ts_tablet_manager.cc:1918] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.812548 1160 log.cc:1199] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:38.812487 1158 consensus_queue.cc:260] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.812878 1160 ts_tablet_manager.cc:1939] T 3cdbdd1e3b734463ab452bafecb75822 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.813656 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1293) took 1495 ms. Trace:
I20250625 07:52:38.813798 1160 rpcz_store.cc:276] 0625 07:52:37.318513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.417078 (+ 98565us) service_pool.cc:225] Handling call
0625 07:52:38.796886 (+1379808us) ts_tablet_manager.cc:1145] Deleting tablet 3cdbdd1e3b734463ab452bafecb75822
0625 07:52:38.796895 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.801933 (+ 5038us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.812299 (+ 10366us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.813592 (+ 1293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369353}
I20250625 07:52:38.815164 1158 ts_tablet_manager.cc:1428] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:38.815459 1158 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.815733 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3cdbdd1e3b734463ab452bafecb75822 (table default.Table_dc62a435b6d44031bc3e5a60a3914e8f [id=f24c46ab73494d2b958f0fa750eadc63]) successfully deleted
I20250625 07:52:38.817149 1158 tablet_bootstrap.cc:654] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.817993 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1329) took 1202 ms. Trace:
I20250625 07:52:38.818142 1160 rpcz_store.cc:276] 0625 07:52:37.615414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.672991 (+ 57577us) service_pool.cc:225] Handling call
0625 07:52:38.817781 (+1144790us) ts_tablet_manager.cc:1145] Deleting tablet 3cc78b37d83b422fae1ed696d45953f6
0625 07:52:38.817798 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.817936 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118196}
W20250625 07:52:38.818765 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3cc78b37d83b422fae1ed696d45953f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc78b37d83b422fae1ed696d45953f6 already in progress: creating tablet
I20250625 07:52:38.820147 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1332) took 1197 ms. Trace:
I20250625 07:52:38.820724 1160 rpcz_store.cc:276] 0625 07:52:37.622823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.699606 (+ 76783us) service_pool.cc:225] Handling call
0625 07:52:38.819952 (+1120346us) ts_tablet_manager.cc:1145] Deleting tablet 0139149bbf574e13be3b0411a83a0593
0625 07:52:38.819966 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.820091 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120054}
W20250625 07:52:38.821406 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0139149bbf574e13be3b0411a83a0593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0139149bbf574e13be3b0411a83a0593 already in progress: creating tablet
I20250625 07:52:38.822074 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1333) took 1128 ms. Trace:
I20250625 07:52:38.822456 1160 rpcz_store.cc:276] 0625 07:52:37.693619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.699912 (+ 6293us) service_pool.cc:225] Handling call
0625 07:52:38.821757 (+1121845us) ts_tablet_manager.cc:1145] Deleting tablet d8ab63bf0a0f4be29cb027505cedf835
0625 07:52:38.821898 (+ 141us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.822010 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116553}
W20250625 07:52:38.823622 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d8ab63bf0a0f4be29cb027505cedf835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ab63bf0a0f4be29cb027505cedf835 already in progress: creating tablet
I20250625 07:52:38.825235 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1331) took 1202 ms. Trace:
I20250625 07:52:38.825381 1160 rpcz_store.cc:276] 0625 07:52:37.622686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.678267 (+ 55581us) service_pool.cc:225] Handling call
0625 07:52:38.824014 (+1145747us) ts_tablet_manager.cc:1145] Deleting tablet 9f30a8b59beb4311aca534ed322aff03
0625 07:52:38.825035 (+ 1021us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.825201 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118622}
I20250625 07:52:38.825729 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:38.826218 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9f30a8b59beb4311aca534ed322aff03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a8b59beb4311aca534ed322aff03 already in progress: creating tablet
I20250625 07:52:38.826342 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffd98b2877b944f0bd484efa2a38a00a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:38.826694 912 raft_consensus.cc:3058] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.831501 912 raft_consensus.cc:2466] T ffd98b2877b944f0bd484efa2a38a00a P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:38.834069 1161 ts_tablet_manager.cc:1918] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.836573 1045 raft_consensus.cc:3058] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:38.838627 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a13381cae324e9494a2497d0190a868 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.839226 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a91007503ad43cc8f6fae30e4206df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:38.839401 756 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d308b0b16a4ab6a3a3589087c94136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:38.839589 713 consensus_peers.cc:487] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:38.840193 713 consensus_peers.cc:487] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:38.840835 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet da895c90d6ee40379fe8181c2dbd7a1e (table default.Table_444c92729289455eaf1a979360c97e5e [id=6fec84e2fefb4a738bce54d790e73177]) successfully deleted
W20250625 07:52:38.841188 713 leader_election.cc:343] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.841333 713 leader_election.cc:304] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.837828 1158 tablet_bootstrap.cc:492] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.842360 1158 ts_tablet_manager.cc:1397] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20250625 07:52:38.843084 1158 raft_consensus.cc:357] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.843214 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1330) took 1220 ms. Trace:
I20250625 07:52:38.843258 1158 raft_consensus.cc:383] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.843322 1158 raft_consensus.cc:738] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.843343 1160 rpcz_store.cc:276] 0625 07:52:37.622568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.674213 (+ 51645us) service_pool.cc:225] Handling call
0625 07:52:38.826617 (+1152404us) ts_tablet_manager.cc:1145] Deleting tablet a1a354a11c894b8f992f3d3cdd66cc06
0625 07:52:38.843032 (+ 16415us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.843163 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1084645}
I20250625 07:52:38.843710 1160 tablet_replica.cc:331] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.843921 1160 raft_consensus.cc:2241] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.844102 1160 raft_consensus.cc:2270] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.844766 1160 ts_tablet_manager.cc:1905] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.844807 1158 consensus_queue.cc:260] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.845496 1162 tablet_bootstrap.cc:492] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.845674 1162 ts_tablet_manager.cc:1397] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20250625 07:52:38.845746 1158 ts_tablet_manager.cc:1428] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:38.846024 1158 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.847653 1158 tablet_bootstrap.cc:654] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:38.848975 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a1a354a11c894b8f992f3d3cdd66cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a354a11c894b8f992f3d3cdd66cc06 already in progress: creating tablet
I20250625 07:52:38.849766 1161 log.cc:1199] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/bfacdde6a8ff4b6eac40d83c1d5b21f6
I20250625 07:52:38.850286 1161 ts_tablet_manager.cc:1939] T bfacdde6a8ff4b6eac40d83c1d5b21f6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.852797 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet bfacdde6a8ff4b6eac40d83c1d5b21f6 (table default.Table_e57f684466224dd5a4a8ff79408e1a4d [id=3dcf0706033c48ceaa1fdd5763284bf0]) successfully deleted
W20250625 07:52:38.853063 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet abb4d9dc2da7453d8336d5589dda3a74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abb4d9dc2da7453d8336d5589dda3a74 already in progress: creating tablet
W20250625 07:52:38.854218 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 53bf5c84b12d4bb399c16eb6e77a28ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bf5c84b12d4bb399c16eb6e77a28ef already in progress: creating tablet
I20250625 07:52:38.856351 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1536933d04014446ac8be0a896f0e6f3
I20250625 07:52:38.857209 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_83d3b9777ac5429ab03a24bddd76bc36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.857960 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83d3b9777ac5429ab03a24bddd76bc36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.859897 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ec1e58fe6d72451682bd47d7238560b6
I20250625 07:52:38.860718 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_818abe262b4b41a4877229ae67d355b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.859428 1160 ts_tablet_manager.cc:1918] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.861294 1161 tablet_replica.cc:331] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:38.861310 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_818abe262b4b41a4877229ae67d355b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.861351 1160 log.cc:1199] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1973678168a84875b747897f08dd1bad
I20250625 07:52:38.861497 1161 raft_consensus.cc:2241] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.861662 1161 raft_consensus.cc:2270] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.861742 1160 ts_tablet_manager.cc:1939] T 1973678168a84875b747897f08dd1bad P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.862207 1161 ts_tablet_manager.cc:1905] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.863073 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a5c64e4e538143159ab5e58f3eb602af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.863529 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5c64e4e538143159ab5e58f3eb602af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.864799 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1c96cdddd3443b8b67f57f4d6c4846e consensus state has no leader
I20250625 07:52:38.865127 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c121aa9cbde14c6bb4a4bcb574484208" } modify_external_catalogs: true
I20250625 07:52:38.865268 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c121aa9cbde14c6bb4a4bcb574484208" } modify_external_catalogs: true
I20250625 07:52:38.789219 2407 pending_rounds.cc:77] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432268258770944 op_type: NO_OP noop_request { }
I20250625 07:52:38.866083 1160 tablet_replica.cc:331] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.866137 2407 raft_consensus.cc:2887] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:38.866235 2407 raft_consensus.cc:2270] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.866335 1160 raft_consensus.cc:2241] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.866372 2498 raft_consensus.cc:2747] T 1796a3305cb94ccb9a0596643a736a96 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.866428 588 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_76ed190456c04c89956f625d4654f801 [id=5523889320fc4a09b49b5e365a722575] alter complete (version 1)
I20250625 07:52:38.866489 1160 raft_consensus.cc:2270] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.866545 2531 tablet.cc:1722] T 1405e55703324bc7a1e26fe2c349bda7 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:38.866979 1160 ts_tablet_manager.cc:1905] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:38.866984 713 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.867177 713 leader_election.cc:304] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:38.867501 2494 raft_consensus.cc:2747] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.868721 1162 raft_consensus.cc:357] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.869036 1162 raft_consensus.cc:383] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.869163 1162 raft_consensus.cc:738] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.869686 1162 consensus_queue.cc:260] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.869801 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet dbdef039d3f842e5957c5bab85bedd42
I20250625 07:52:38.870187 590 catalog_manager.cc:6652] ReplaceTablet: tablet dbdef039d3f842e5957c5bab85bedd42 deleted and replaced by tablet ae1300de7b364698935bdebdc6727c4a
I20250625 07:52:38.870738 1162 ts_tablet_manager.cc:1428] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250625 07:52:38.871255 1162 tablet_bootstrap.cc:492] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.872519 883 tablet_service.cc:1515] Processing DeleteTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.872596 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1536933d04014446ac8be0a896f0e6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.872519 879 tablet_service.cc:1515] Processing DeleteTablet for tablet ec1e58fe6d72451682bd47d7238560b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.872939 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1973678168a84875b747897f08dd1bad (table default.Table_83f5d615c53042dd888e74c415cc3caf [id=1f730d2bc75b4fab86e26a382a94ce2c]) successfully deleted
I20250625 07:52:38.873342 879 tablet_service.cc:1515] Processing DeleteTablet for tablet dbdef039d3f842e5957c5bab85bedd42 with delete_type TABLET_DATA_DELETED (replaced by tablet ae1300de7b364698935bdebdc6727c4a) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:38.873416 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d97f4c56eb574432af661966fcda939b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97f4c56eb574432af661966fcda939b already in progress: creating tablet
I20250625 07:52:38.875264 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 1536933d04014446ac8be0a896f0e6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.875370 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet ec1e58fe6d72451682bd47d7238560b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.875538 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet dbdef039d3f842e5957c5bab85bedd42 with delete_type TABLET_DATA_DELETED (replaced by tablet ae1300de7b364698935bdebdc6727c4a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.876533 1158 tablet_bootstrap.cc:492] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.876677 1158 ts_tablet_manager.cc:1397] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s
I20250625 07:52:38.876900 2407 ts_tablet_manager.cc:1905] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.877585 1158 raft_consensus.cc:357] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.877807 1158 raft_consensus.cc:383] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.877872 1158 raft_consensus.cc:738] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.878067 1158 consensus_queue.cc:260] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.879422 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet f04016cda7f746d49e30be3b1c9c18b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.879514 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 110fe299c7c44df69f459da72a1138a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.879681 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e5a207d515449e298936d8bb532437a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.880795 1163 tablet_bootstrap.cc:492] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:38.881000 1163 ts_tablet_manager.cc:1397] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.111s user 0.027s sys 0.015s
I20250625 07:52:38.881404 1162 tablet_bootstrap.cc:654] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.882133 1163 raft_consensus.cc:357] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.882397 1163 raft_consensus.cc:383] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.882364 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
I20250625 07:52:38.882531 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:38.882691 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:38.882943 1163 raft_consensus.cc:738] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.883283 1163 consensus_queue.cc:260] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.884003 1163 ts_tablet_manager.cc:1428] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:38.884361 1163 tablet_bootstrap.cc:492] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:38.884495 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.886009 1163 tablet_bootstrap.cc:654] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.887522 1160 ts_tablet_manager.cc:1918] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.888618 1160 log.cc:1199] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/08cd482ceec244c2bed2b116d5c14d89
I20250625 07:52:38.889081 1160 ts_tablet_manager.cc:1939] T 08cd482ceec244c2bed2b116d5c14d89 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.890010 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 08cd482ceec244c2bed2b116d5c14d89 (table default.Table_21de3cd434744102b0d5ffbbc17c18c3 [id=f129efbf6ddb47d1a59e7f511637654a]) successfully deleted
I20250625 07:52:38.891038 1160 tablet_replica.cc:331] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.891291 1160 raft_consensus.cc:2241] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.891549 1160 raft_consensus.cc:2270] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.892194 2494 raft_consensus.cc:491] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.892342 2494 raft_consensus.cc:513] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.892506 1160 ts_tablet_manager.cc:1905] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.893143 2494 leader_election.cc:290] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.885587 713 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.895521 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_00bcc156be73401ca72064b265fc6c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.896104 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00bcc156be73401ca72064b265fc6c2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.893942 713 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.897686 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_37c181e25508461b927d103f11855cf6" } modify_external_catalogs: true
W20250625 07:52:38.897779 713 leader_election.cc:343] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.897840 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_37c181e25508461b927d103f11855cf6" } modify_external_catalogs: true
W20250625 07:52:38.897941 713 leader_election.cc:343] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:38.900894 1158 ts_tablet_manager.cc:1428] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20250625 07:52:38.901199 1158 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.902865 1158 tablet_bootstrap.cc:654] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.910879 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_045259525d4b454fbc351c409b78e67a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.911464 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_045259525d4b454fbc351c409b78e67a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.913527 986 connection.cc:537] client connection to 127.0.104.61:43317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:38.913977 851 connection.cc:537] client connection to 127.0.104.61:43317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:52:38.918605 879 tablet_service.cc:1515] Processing DeleteTablet for tablet e181a3f1ae2c49349718d842c166b1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:38.926331 1158 tablet_bootstrap.cc:492] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.926492 1158 ts_tablet_manager.cc:1397] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20250625 07:52:38.929067 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet abb4d9dc2da7453d8336d5589dda3a74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.927238 1158 raft_consensus.cc:357] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.930828 1158 raft_consensus.cc:383] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.930893 1158 raft_consensus.cc:738] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.930581 1161 ts_tablet_manager.cc:1918] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.931219 1161 log.cc:1199] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c50c35cac8e94e088cdacb664b8adfe1
I20250625 07:52:38.931679 1161 ts_tablet_manager.cc:1939] T c50c35cac8e94e088cdacb664b8adfe1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:38.931880 1158 consensus_queue.cc:260] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.933082 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c50c35cac8e94e088cdacb664b8adfe1 (table default.Table_f6e3be4354c248d8abb373c9f0b5c6c9 [id=860ff357dc3a4a2dad9d850c113cf464]) successfully deleted
I20250625 07:52:38.937420 1158 ts_tablet_manager.cc:1428] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250625 07:52:38.937759 1158 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:38.940981 2506 raft_consensus.cc:491] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.941151 2506 raft_consensus.cc:513] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.941376 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ca9bb00cf9834f0797220f17c7ad930d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:38.941711 2506 leader_election.cc:290] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:38.941958 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca9bb00cf9834f0797220f17c7ad930d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:38.942389 1160 ts_tablet_manager.cc:1918] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.942502 1160 log.cc:1199] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/44bfcc2b20ba4d0b83641988f2c0ee35
I20250625 07:52:38.942793 1160 ts_tablet_manager.cc:1939] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.943190 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7b42526a3fb543b894d40b54df370f55" } new_table_name: "default.Table_6b271657b9bb4778a7ef4b5184b245d0" modify_external_catalogs: true
W20250625 07:52:38.944555 985 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 547b758c2e6c4558bb232f14b37aa81a
W20250625 07:52:38.945057 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 05991b31ebf945f29fca8c060b21e7a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05991b31ebf945f29fca8c060b21e7a7 already in progress: creating tablet
W20250625 07:52:38.945113 714 connection.cc:537] client connection to 127.0.104.61:43317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:38.945459 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4709f33f040149c49bd273524c9fd4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4709f33f040149c49bd273524c9fd4f0 already in progress: creating tablet
I20250625 07:52:38.949066 1161 tablet_replica.cc:331] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.949342 1161 raft_consensus.cc:2241] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.949570 1161 raft_consensus.cc:2270] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.949872 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1336) took 1047 ms. Trace:
I20250625 07:52:38.950012 1160 rpcz_store.cc:276] 0625 07:52:37.902656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.904881 (+ 2225us) service_pool.cc:225] Handling call
0625 07:52:38.890324 (+985443us) ts_tablet_manager.cc:1145] Deleting tablet 44bfcc2b20ba4d0b83641988f2c0ee35
0625 07:52:38.890961 (+ 637us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.895783 (+ 4822us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.942251 (+ 46468us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.949832 (+ 7581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":985288}
I20250625 07:52:38.950456 1160 tablet_replica.cc:331] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.950654 1160 raft_consensus.cc:2241] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.950806 1160 raft_consensus.cc:2270] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:38.951161 1158 tablet_bootstrap.cc:654] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.951447 1160 ts_tablet_manager.cc:1905] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.951416 1161 ts_tablet_manager.cc:1905] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.952224 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 44bfcc2b20ba4d0b83641988f2c0ee35 (table default.Table_9f2673ddf3e34a969d58c4514580bdcd [id=2aad9533753e4ff9b9c0bab636a8f85b]) successfully deleted
W20250625 07:52:38.952522 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3f1f025775d04da09eda17af4ac4fe10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f1f025775d04da09eda17af4ac4fe10 already in progress: creating tablet
I20250625 07:52:38.955222 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dd0ca292d8b248ce8ddf004badf3ca25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:38.955799 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd0ca292d8b248ce8ddf004badf3ca25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:38.957126 584 connection.cc:537] client connection to 127.0.104.61:43317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:52:38.957229 584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20250625 07:52:38.958285 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:38.958606 914 raft_consensus.cc:2466] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:38.959317 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet faf049714a7e4d7eb947c29f272bbb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:38.960302 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.961129 1162 tablet_bootstrap.cc:492] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:38.964972 1160 ts_tablet_manager.cc:1918] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.965101 1160 log.cc:1199] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/065822c50586488785f7d9d53841a4dd
I20250625 07:52:38.965418 1160 ts_tablet_manager.cc:1939] T 065822c50586488785f7d9d53841a4dd P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.966219 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1337) took 1063 ms. Trace:
I20250625 07:52:38.966325 1160 rpcz_store.cc:276] 0625 07:52:37.902844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.905063 (+ 2219us) service_pool.cc:225] Handling call
0625 07:52:38.950399 (+1045336us) ts_tablet_manager.cc:1145] Deleting tablet 065822c50586488785f7d9d53841a4dd
0625 07:52:38.950413 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.961569 (+ 11156us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.964846 (+ 3277us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:38.966200 (+ 1354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045191}
I20250625 07:52:38.966950 1160 tablet_replica.cc:331] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.967182 1160 raft_consensus.cc:2241] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.967365 1160 pending_rounds.cc:70] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Trying to abort 1 pending ops.
I20250625 07:52:38.967469 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 065822c50586488785f7d9d53841a4dd (table default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]) successfully deleted
I20250625 07:52:38.967523 1160 pending_rounds.cc:77] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7171432275222630400 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b" dest_uuid: "191b611b772a4fda9d1cba98776138a7" new_extra_config { } }
I20250625 07:52:38.967891 1160 raft_consensus.cc:2270] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:38.967885 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 53bf5c84b12d4bb399c16eb6e77a28ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bf5c84b12d4bb399c16eb6e77a28ef already in progress: creating tablet
W20250625 07:52:38.968307 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d97f4c56eb574432af661966fcda939b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97f4c56eb574432af661966fcda939b already in progress: creating tablet
I20250625 07:52:38.970885 1161 ts_tablet_manager.cc:1918] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:38.970919 1162 ts_tablet_manager.cc:1397] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20250625 07:52:38.972677 1162 raft_consensus.cc:357] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.975872 1160 ts_tablet_manager.cc:1905] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.978631 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_462e9992625a498e814ed911f3ee61a6" } modify_external_catalogs: true
I20250625 07:52:38.978849 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_462e9992625a498e814ed911f3ee61a6" } modify_external_catalogs: true
I20250625 07:52:38.974287 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea62dfb8f20f4caeaa79ea1e85d99930" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:38.980511 1158 tablet_bootstrap.cc:492] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:38.982110 1158 ts_tablet_manager.cc:1397] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.003s
I20250625 07:52:38.983013 1158 raft_consensus.cc:357] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.983270 1158 raft_consensus.cc:383] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:38.983371 1158 raft_consensus.cc:738] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:38.983624 1158 consensus_queue.cc:260] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:38.984431 1160 ts_tablet_manager.cc:1918] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:38.984555 1160 log.cc:1199] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/abb4d9dc2da7453d8336d5589dda3a74
W20250625 07:52:38.985107 2584 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 07:52:38.915706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:43317 (local address 127.0.104.2:43113)
0625 07:52:38.984671 (+ 68965us) negotiation.cc:107] Waiting for socket to connect
0625 07:52:38.984841 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":68823,"threads_started":1}
I20250625 07:52:38.986640 1158 ts_tablet_manager.cc:1428] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:38.987053 1158 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:38.987130 2585 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 07:52:38.920381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:43317 (local address 127.0.104.3:35599)
0625 07:52:38.986279 (+ 65898us) negotiation.cc:107] Waiting for socket to connect
0625 07:52:38.986675 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":65741,"threads_started":1}
I20250625 07:52:38.991922 1160 ts_tablet_manager.cc:1939] T abb4d9dc2da7453d8336d5589dda3a74 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:38.992013 1158 tablet_bootstrap.cc:654] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:38.992794 1161 log.cc:1199] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/bbe5c37fc9bd49c685de8f469a7af0c6
I20250625 07:52:38.993188 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b [id=c80e78f1ef124af2b8f405e7d162ade8]) successfully deleted
I20250625 07:52:38.993498 1160 tablet_replica.cc:331] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:38.993542 1161 ts_tablet_manager.cc:1939] T bbe5c37fc9bd49c685de8f469a7af0c6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:38.993628 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 05991b31ebf945f29fca8c060b21e7a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05991b31ebf945f29fca8c060b21e7a7 already in progress: creating tablet
I20250625 07:52:38.993697 1160 raft_consensus.cc:2241] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.993841 1160 raft_consensus.cc:2270] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:38.993983 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4709f33f040149c49bd273524c9fd4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4709f33f040149c49bd273524c9fd4f0 already in progress: creating tablet
W20250625 07:52:38.994251 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3f1f025775d04da09eda17af4ac4fe10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f1f025775d04da09eda17af4ac4fe10 already in progress: creating tablet
I20250625 07:52:38.994371 1160 ts_tablet_manager.cc:1905] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:38.994527 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet bbe5c37fc9bd49c685de8f469a7af0c6 (table default.Table_e75429a686be4882a9676251755c548b [id=c1ded4dcad0e4e0cb3cb7937c02353c1]) successfully deleted
W20250625 07:52:38.997529 986 leader_election.cc:343] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 547b758c2e6c4558bb232f14b37aa81a
I20250625 07:52:38.997567 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3488f79d09054f6dbfc6b54c1dee0d6c
I20250625 07:52:38.997741 986 leader_election.cc:304] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:38.998100 2506 raft_consensus.cc:2747] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:38.998500 1161 tablet_replica.cc:331] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:38.998569 2475 raft_consensus.cc:491] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:38.998692 2475 raft_consensus.cc:513] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:38.999022 1161 raft_consensus.cc:2241] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:38.999202 2475 leader_election.cc:290] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.000489 1160 ts_tablet_manager.cc:1918] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.000497 2475 raft_consensus.cc:491] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.000613 1160 log.cc:1199] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a40dacaa394747aaa100a0527e14eed0
I20250625 07:52:39.000628 2475 raft_consensus.cc:513] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.000913 1161 raft_consensus.cc:2270] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.000993 1160 ts_tablet_manager.cc:1939] T a40dacaa394747aaa100a0527e14eed0 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.001266 2475 leader_election.cc:290] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:38.999719 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.001806 850 leader_election.cc:343] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.002125 1160 tablet_replica.cc:331] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.002298 1160 raft_consensus.cc:2241] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.002435 1160 raft_consensus.cc:2270] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.002403 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.002915 1160 ts_tablet_manager.cc:1905] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:39.002938 850 leader_election.cc:343] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.003830 1163 tablet_bootstrap.cc:492] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.003945 1163 ts_tablet_manager.cc:1397] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.040s
I20250625 07:52:39.004667 1163 raft_consensus.cc:357] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.004840 1163 raft_consensus.cc:383] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.004887 1163 raft_consensus.cc:738] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.005079 1163 consensus_queue.cc:260] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.005437 1161 ts_tablet_manager.cc:1905] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.005573 1163 ts_tablet_manager.cc:1428] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:39.005833 1163 tablet_bootstrap.cc:492] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.007192 1163 tablet_bootstrap.cc:654] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.007601 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.007845 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.008570 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a40dacaa394747aaa100a0527e14eed0 (table default.Table_f801c447326646d1ab93ac87fff7843a [id=cf2c5d2f59bc462bb1fe081f619e5841]) successfully deleted
W20250625 07:52:39.008989 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e8fa349c737f4cebb1225b20e62e3dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8fa349c737f4cebb1225b20e62e3dcb already in progress: creating tablet
W20250625 07:52:39.009310 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7d06e9aae7eb408c99d97e52fc9f691c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d06e9aae7eb408c99d97e52fc9f691c already in progress: creating tablet
I20250625 07:52:39.009743 1162 raft_consensus.cc:383] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.009927 1162 raft_consensus.cc:738] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.010243 1162 consensus_queue.cc:260] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.010910 1162 ts_tablet_manager.cc:1428] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250625 07:52:39.010924 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b8f97eac18cc433b8871e352302a53c7" } new_table_name: "default.Table_1398370c6d1d42e784eeb2cad1df0cac" modify_external_catalogs: true
I20250625 07:52:39.011301 1162 tablet_bootstrap.cc:492] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:38.982175 713 leader_election.cc:304] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.012988 2494 raft_consensus.cc:2802] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:39.013077 2494 raft_consensus.cc:491] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:39.013134 2494 raft_consensus.cc:3058] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:39.014153 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e8fa349c737f4cebb1225b20e62e3dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8fa349c737f4cebb1225b20e62e3dcb already in progress: creating tablet
W20250625 07:52:39.014389 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7d06e9aae7eb408c99d97e52fc9f691c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d06e9aae7eb408c99d97e52fc9f691c already in progress: creating tablet
W20250625 07:52:39.014575 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 8a13381cae324e9494a2497d0190a868 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a13381cae324e9494a2497d0190a868 already in progress: creating tablet
W20250625 07:52:39.014737 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3a91007503ad43cc8f6fae30e4206df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a91007503ad43cc8f6fae30e4206df8 already in progress: creating tablet
W20250625 07:52:39.014894 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b4d308b0b16a4ab6a3a3589087c94136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4d308b0b16a4ab6a3a3589087c94136 already in progress: creating tablet
I20250625 07:52:39.015723 1160 ts_tablet_manager.cc:1918] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.015831 1160 log.cc:1199] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/a6c0834adf1748c6a67805d26b973171
I20250625 07:52:39.015745 2494 raft_consensus.cc:513] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.016100 1160 ts_tablet_manager.cc:1939] T a6c0834adf1748c6a67805d26b973171 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.016418 2494 leader_election.cc:290] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:39.017962 713 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 716e0c4dab584788bbb7631b4ca934bf
I20250625 07:52:39.018234 713 leader_election.cc:304] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.018591 2494 raft_consensus.cc:2747] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.022297 1161 ts_tablet_manager.cc:1918] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:39.023461 584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317): Couldn't send request to peer 789c06e7de2842a1b40e97e0778ab6c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:39.024102 1163 tablet_bootstrap.cc:492] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.024389 1163 ts_tablet_manager.cc:1397] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20250625 07:52:39.025442 1163 raft_consensus.cc:357] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.025684 1163 raft_consensus.cc:383] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.025815 1163 raft_consensus.cc:738] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.026091 1163 consensus_queue.cc:260] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.026823 1163 ts_tablet_manager.cc:1428] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:39.027189 1163 tablet_bootstrap.cc:492] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:38.988232 2586 negotiation.cc:337] Failed RPC negotiation. Trace:
0625 07:52:38.947257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:43317 (local address 127.0.104.1:33407)
0625 07:52:38.987786 (+ 40529us) negotiation.cc:107] Waiting for socket to connect
0625 07:52:38.987904 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40369,"threads_started":1}
I20250625 07:52:39.000284 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.028892 1163 tablet_bootstrap.cc:654] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.029683 1160 tablet_replica.cc:331] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.030515 1160 raft_consensus.cc:2241] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.030695 1160 raft_consensus.cc:2270] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.031396 1160 ts_tablet_manager.cc:1905] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:39.032992 851 leader_election.cc:343] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.035013 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet a6c0834adf1748c6a67805d26b973171 (table default.Table_1444a3af50a54dfe9739e5c255dc27a4 [id=8e2d9c1cd8cd4b149d3be543ab086667]) successfully deleted
W20250625 07:52:39.036631 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 8a13381cae324e9494a2497d0190a868 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a13381cae324e9494a2497d0190a868 already in progress: creating tablet
I20250625 07:52:39.025869 1158 tablet_bootstrap.cc:492] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.036877 1158 ts_tablet_manager.cc:1397] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.001s
I20250625 07:52:39.037746 1158 raft_consensus.cc:357] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.037933 1158 raft_consensus.cc:383] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.037997 1158 raft_consensus.cc:738] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.038206 1158 consensus_queue.cc:260] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.038697 1158 ts_tablet_manager.cc:1428] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250625 07:52:39.039776 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 3a91007503ad43cc8f6fae30e4206df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a91007503ad43cc8f6fae30e4206df8 already in progress: creating tablet
W20250625 07:52:39.040269 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b4d308b0b16a4ab6a3a3589087c94136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4d308b0b16a4ab6a3a3589087c94136 already in progress: creating tablet
I20250625 07:52:39.041985 851 leader_election.cc:304] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.042272 2475 raft_consensus.cc:2747] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.037467 1160 ts_tablet_manager.cc:1918] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.042757 1160 log.cc:1199] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/dddbc1200acf4e7fa93ef85b398d5832
I20250625 07:52:39.043079 1160 ts_tablet_manager.cc:1939] T dddbc1200acf4e7fa93ef85b398d5832 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.044440 1160 tablet_replica.cc:331] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.044638 1160 raft_consensus.cc:2241] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.044801 1160 raft_consensus.cc:2270] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.041501 1163 tablet_bootstrap.cc:492] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.045053 1163 ts_tablet_manager.cc:1397] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250625 07:52:39.045599 1158 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.045742 1163 raft_consensus.cc:357] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.045981 1163 raft_consensus.cc:383] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.046085 1163 raft_consensus.cc:738] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.046348 1163 consensus_queue.cc:260] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.046904 1163 ts_tablet_manager.cc:1428] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:39.047252 1163 tablet_bootstrap.cc:492] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.047310 1158 tablet_bootstrap.cc:654] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.047659 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1dc8dbee6e0e40b7af3efcb9fdea2d38" } new_table_name: "default.Table_a223b905e0554e24bc19236aa2cacb05" modify_external_catalogs: true
I20250625 07:52:39.048745 1163 tablet_bootstrap.cc:654] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.049134 1161 log.cc:1199] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/547b758c2e6c4558bb232f14b37aa81a
I20250625 07:52:39.049798 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 086c9321a70e44b5bb3ea419bb466fbe with delete_type TABLET_DATA_DELETED (replaced by tablet 0bfa1c4d4b6e4530892118189a5ab516) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.049939 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 97b9a14151fc472e956126620633be32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.050252 1161 ts_tablet_manager.cc:1939] T 547b758c2e6c4558bb232f14b37aa81a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:39.050614 713 leader_election.cc:304] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.052089 2494 raft_consensus.cc:2802] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.052376 2494 raft_consensus.cc:695] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.052564 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet dddbc1200acf4e7fa93ef85b398d5832 (table default.Table_c62e57a0d535414a93d82dca7dd477ba [id=1dbec332cb8a45eb8b6114bd1b9bdd5a]) successfully deleted
I20250625 07:52:39.052733 1161 tablet_replica.cc:331] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:39.052763 2494 consensus_queue.cc:237] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.052850 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1536933d04014446ac8be0a896f0e6f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1536933d04014446ac8be0a896f0e6f3 already in progress: creating tablet
W20250625 07:52:39.053069 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ec1e58fe6d72451682bd47d7238560b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1e58fe6d72451682bd47d7238560b6 already in progress: creating tablet
I20250625 07:52:39.053170 1161 raft_consensus.cc:2241] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:39.053264 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet dbdef039d3f842e5957c5bab85bedd42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbdef039d3f842e5957c5bab85bedd42 already in progress: creating tablet
I20250625 07:52:39.053642 1161 raft_consensus.cc:2270] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.055315 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 60c01f0a75674c469b41fac638799afa
I20250625 07:52:39.055440 713 leader_election.cc:304] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.055763 2494 raft_consensus.cc:2802] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.056012 2494 raft_consensus.cc:695] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.056134 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5ab142d37cc64674ba7522485ec90897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.056304 2494 consensus_queue.cc:237] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.056857 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ab142d37cc64674ba7522485ec90897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.004751 2407 ts_tablet_manager.cc:1918] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:39.005403 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.059257 2407 log.cc:1199] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9bccda893fd5466886a4d253172309e1
W20250625 07:52:39.059578 851 leader_election.cc:343] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.059708 2407 ts_tablet_manager.cc:1939] T 9bccda893fd5466886a4d253172309e1 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:39.059790 851 leader_election.cc:304] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.060464 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1345) took 1782 ms. Trace:
I20250625 07:52:39.060544 2407 rpcz_store.cc:276] 0625 07:52:37.277637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.277679 (+ 42us) service_pool.cc:225] Handling call
0625 07:52:38.788433 (+1510754us) ts_tablet_manager.cc:1145] Deleting tablet 9bccda893fd5466886a4d253172309e1
0625 07:52:38.788447 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:38.909330 (+120883us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.004610 (+ 95280us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.060446 (+ 55836us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9987,"tablet-delete.queue_time_us":1510581}
I20250625 07:52:39.060762 2475 raft_consensus.cc:2747] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.061730 1158 tablet_bootstrap.cc:492] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.061904 1158 ts_tablet_manager.cc:1397] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250625 07:52:39.062330 1161 ts_tablet_manager.cc:1905] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.063040 1158 raft_consensus.cc:357] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.063266 1158 raft_consensus.cc:383] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.063335 1158 raft_consensus.cc:738] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.063573 1158 consensus_queue.cc:260] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.066578 1163 tablet_bootstrap.cc:492] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.068358 1163 ts_tablet_manager.cc:1397] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250625 07:52:39.069123 1163 raft_consensus.cc:357] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.069401 1158 ts_tablet_manager.cc:1428] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:39.069628 1163 raft_consensus.cc:383] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.069871 1158 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.070452 1163 raft_consensus.cc:738] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.070746 1163 consensus_queue.cc:260] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.071316 1163 ts_tablet_manager.cc:1428] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:39.071606 1163 tablet_bootstrap.cc:492] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.073280 1163 tablet_bootstrap.cc:654] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.068955 2562 tablet.cc:1717] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:39.073722 2407 tablet_replica.cc:331] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:39.073915 2407 raft_consensus.cc:2241] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.074082 2407 raft_consensus.cc:2270] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.074589 2407 ts_tablet_manager.cc:1905] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.069202 1162 tablet_bootstrap.cc:654] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.079820 1158 tablet_bootstrap.cc:654] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.093816 1158 tablet_bootstrap.cc:492] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.093966 1158 ts_tablet_manager.cc:1397] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250625 07:52:39.094740 1158 raft_consensus.cc:357] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.094913 1158 raft_consensus.cc:383] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.094952 1158 raft_consensus.cc:738] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.095155 1158 consensus_queue.cc:260] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.097214 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 547b758c2e6c4558bb232f14b37aa81a (table default.Table_0c0f428f2b54479a8e0bd1f36ef2dede [id=83522331e5df4bafaa65c81be08c7525]) successfully deleted
I20250625 07:52:39.097922 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c01f0a75674c469b41fac638799afa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.097929 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c01f0a75674c469b41fac638799afa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.098680 1160 ts_tablet_manager.cc:1905] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.098858 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:33894 (request call id 1403) took 1305 ms. Trace:
I20250625 07:52:39.098950 2562 rpcz_store.cc:276] 0625 07:52:37.792943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.793148 (+ 205us) service_pool.cc:225] Handling call
0625 07:52:39.098836 (+1305688us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:37.818581 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 07:52:37.818661 (+ 80us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 07:52:37.880543 (+ 61882us) alter_schema_op.cc:66] Acquired schema lock
0625 07:52:37.880558 (+ 15us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0625 07:52:37.880596 (+ 38us) alter_schema_op.cc:128] START. Timestamp: 1750837957880577
0625 07:52:37.880600 (+ 4us) op_driver.cc:348] REPLICATION: starting
0625 07:52:37.880783 (+ 183us) log.cc:844] Serialized 173 byte log entry
0625 07:52:39.067844 (+1187061us) op_driver.cc:464] REPLICATION: finished
0625 07:52:39.068919 (+ 1075us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0625 07:52:39.073793 (+ 4874us) log.cc:844] Serialized 94 byte log entry
0625 07:52:39.098789 (+ 24996us) alter_schema_op.cc:73] Released schema lock
0625 07:52:39.098795 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":997,"prepare.queue_time_us":13127,"prepare.run_cpu_time_us":10117,"prepare.run_wall_time_us":62256,"replication_time_us":1187211,"thread_start_us":37965,"threads_started":2,"wal-append.queue_time_us":24671}]]}
I20250625 07:52:39.099762 2562 tablet.cc:1722] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:39.105034 1158 ts_tablet_manager.cc:1428] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250625 07:52:39.106040 1158 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.110900 2562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:33894 (request call id 1402) took 1345 ms. Trace:
I20250625 07:52:39.111057 2562 rpcz_store.cc:276] 0625 07:52:37.765093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.765164 (+ 71us) service_pool.cc:225] Handling call
0625 07:52:39.110873 (+1345709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0625 07:52:37.817882 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0625 07:52:37.817993 (+ 111us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0625 07:52:37.818002 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0625 07:52:37.818010 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0625 07:52:37.818051 (+ 41us) alter_schema_op.cc:128] START. Timestamp: 1750837957818025
0625 07:52:37.818054 (+ 3us) op_driver.cc:348] REPLICATION: starting
0625 07:52:37.818247 (+ 193us) log.cc:844] Serialized 173 byte log entry
0625 07:52:39.078613 (+1260366us) op_driver.cc:464] REPLICATION: finished
0625 07:52:39.099742 (+ 21129us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0625 07:52:39.100139 (+ 397us) log.cc:844] Serialized 5 byte log entry
0625 07:52:39.110173 (+ 10034us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.110774 (+ 601us) log.cc:844] Serialized 14 byte log entry
0625 07:52:39.110818 (+ 44us) alter_schema_op.cc:73] Released schema lock
0625 07:52:39.110825 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21076,"prepare.queue_time_us":25085,"prepare.run_cpu_time_us":2169,"prepare.run_wall_time_us":3767,"raft.queue_time_us":43,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":1260524,"thread_start_us":35368,"threads_started":4,"wal-append.queue_time_us":10705,"wal-append.run_cpu_time_us":487,"wal-append.run_wall_time_us":1001122}]]}
I20250625 07:52:39.125635 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aada5cc96edc434b8e666d538d4eb5e8" } modify_external_catalogs: true
I20250625 07:52:39.125855 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aada5cc96edc434b8e666d538d4eb5e8" } modify_external_catalogs: true
I20250625 07:52:39.127579 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d393a85f5fde4b3fb57e5b3ec372791c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.131920 2604 raft_consensus.cc:491] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.132230 2604 raft_consensus.cc:513] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.132455 1158 tablet_bootstrap.cc:654] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.131919 2562 tablet.cc:1722] T 4da889781dbd41aebac86e9833cf2b7e P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:39.133816 2604 leader_election.cc:290] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.134694 2604 raft_consensus.cc:491] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.134892 2604 raft_consensus.cc:513] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.135433 2604 leader_election.cc:290] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.136958 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a1e2434d6f04ac099d67bdc3ebdf335 consensus state has no leader
I20250625 07:52:39.137357 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d25be557c62b486592bc6f4c1025354e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.137892 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d25be557c62b486592bc6f4c1025354e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:39.139389 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5c0d5b5da514c57af4266524bdf4f98 consensus state has no leader
W20250625 07:52:39.139545 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 61277.9us
I20250625 07:52:39.139757 713 leader_election.cc:304] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:39.139735 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b2d591e4706149daa6e50d9466ea91ea" } new_table_name: "default.Table_1285c71f306940e6bad0c7a7e6422d0b" modify_external_catalogs: true
W20250625 07:52:39.140102 713 leader_election.cc:343] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: ea62dfb8f20f4caeaa79ea1e85d99930
I20250625 07:52:39.140792 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 1536933d04014446ac8be0a896f0e6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.140992 746 tablet_service.cc:1515] Processing DeleteTablet for tablet ec1e58fe6d72451682bd47d7238560b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.141160 742 tablet_service.cc:1515] Processing DeleteTablet for tablet dbdef039d3f842e5957c5bab85bedd42 with delete_type TABLET_DATA_DELETED (replaced by tablet ae1300de7b364698935bdebdc6727c4a) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:39.141287 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a25f89bffcd0430a976113d512c6a699 consensus state has no leader
I20250625 07:52:39.141613 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3488f79d09054f6dbfc6b54c1dee0d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.141659 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6f19a0a49fd04cffa91026e8f47a8bc7" } modify_external_catalogs: true
I20250625 07:52:39.141813 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 60c01f0a75674c469b41fac638799afa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.141808 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6f19a0a49fd04cffa91026e8f47a8bc7" } modify_external_catalogs: true
I20250625 07:52:39.142107 2599 raft_consensus.cc:2802] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:39.142176 2599 raft_consensus.cc:491] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:39.142269 2599 raft_consensus.cc:3058] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.142666 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c8c69b95a3b14bb496e64b147ad6ad8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.143196 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8c69b95a3b14bb496e64b147ad6ad8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.146595 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f3e9de3fec8145649644fe7e4c9bfd7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.147154 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3e9de3fec8145649644fe7e4c9bfd7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.148577 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3cdbdd1e3b734463ab452bafecb75822" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:39.149197 2599 raft_consensus.cc:513] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.149632 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f4bb74391a1c4e2792226d259a07f977 with delete_type TABLET_DATA_DELETED (replaced by tablet dfc22a46594f47deb8b142bc14c81a0b) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.149914 2599 leader_election.cc:290] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.153515 2534 tablet.cc:1717] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:39.154255 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9bccda893fd5466886a4d253172309e1 (table default.Table_3910e10f6f4b41eeabfd11ac6c4343c7 [id=c90351757d464813b5d0d67936dd69c5]) successfully deleted
I20250625 07:52:39.155303 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.155897 2606 tablet.cc:1722] T 2db37ece7a964f47911fe4fc7434cd07 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:39.155989 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:39.156193 914 raft_consensus.cc:2466] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:39.157222 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d109c9ad2e7d452e9533297a490c2031 consensus state has no leader
I20250625 07:52:39.158982 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe630469fe7c4f4bac28277896eddc9c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.160640 2534 tablet.cc:1722] T 4da889781dbd41aebac86e9833cf2b7e P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:39.162657 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cfcdd655f66c4381b702e1d689f72291"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.162871 1158 tablet_bootstrap.cc:492] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.162990 1158 ts_tablet_manager.cc:1397] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
W20250625 07:52:39.163229 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfcdd655f66c4381b702e1d689f72291 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.163753 1158 raft_consensus.cc:357] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.163902 1158 raft_consensus.cc:383] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.163954 1158 raft_consensus.cc:738] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.164144 1158 consensus_queue.cc:260] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.167450 1162 tablet_bootstrap.cc:492] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:39.167694 1162 ts_tablet_manager.cc:1397] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.000s
I20250625 07:52:39.168632 1158 ts_tablet_manager.cc:1428] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:39.168782 1162 raft_consensus.cc:357] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.169029 1162 raft_consensus.cc:383] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.169137 1162 raft_consensus.cc:738] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.169166 1158 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.169445 1162 consensus_queue.cc:260] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.172027 1158 tablet_bootstrap.cc:654] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.169973 1162 ts_tablet_manager.cc:1428] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:39.173521 1162 tablet_bootstrap.cc:492] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:39.174182 2475 raft_consensus.cc:491] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.174328 2475 raft_consensus.cc:513] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.174428 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:39.174985 2475 leader_election.cc:290] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.175915 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.176527 850 leader_election.cc:343] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.177188 1163 tablet_bootstrap.cc:492] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.177330 1163 ts_tablet_manager.cc:1397] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.000s
I20250625 07:52:39.177312 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:39.177954 851 leader_election.cc:343] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.178064 851 leader_election.cc:304] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.178454 2475 raft_consensus.cc:2747] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.178788 2506 raft_consensus.cc:491] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.178920 2506 raft_consensus.cc:513] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.179450 2506 leader_election.cc:290] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.178072 1163 raft_consensus.cc:357] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.180020 1163 raft_consensus.cc:383] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.180111 1163 raft_consensus.cc:738] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.180331 1163 consensus_queue.cc:260] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.180680 986 leader_election.cc:343] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 97b9a14151fc472e956126620633be32
W20250625 07:52:39.181239 985 leader_election.cc:343] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 97b9a14151fc472e956126620633be32
I20250625 07:52:39.181355 985 leader_election.cc:304] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.181550 2506 raft_consensus.cc:2747] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.181970 1163 ts_tablet_manager.cc:1428] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250625 07:52:39.182235 1163 tablet_bootstrap.cc:492] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.183924 1163 tablet_bootstrap.cc:654] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.186471 1161 ts_tablet_manager.cc:1918] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.186632 1161 log.cc:1199] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/34a338005a7b44b9b0fba7a7096cfcb8
I20250625 07:52:39.186726 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eafc34b6a2f84ee48f05d17454df5bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.187058 1161 ts_tablet_manager.cc:1939] T 34a338005a7b44b9b0fba7a7096cfcb8 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:39.187431 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eafc34b6a2f84ee48f05d17454df5bbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.187997 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 34a338005a7b44b9b0fba7a7096cfcb8 (table default.Table_3aeb5c70494240aaaad67f311ec1723b [id=2cef554f33094c77b8684986968dadef]) successfully deleted
I20250625 07:52:39.188490 2475 raft_consensus.cc:491] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.189451 2475 raft_consensus.cc:513] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.190184 2475 leader_election.cc:290] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.190654 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:39.190851 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e181a3f1ae2c49349718d842c166b1e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.191355 850 leader_election.cc:343] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.192036 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e181a3f1ae2c49349718d842c166b1e6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:39.192498 851 leader_election.cc:343] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.192589 851 leader_election.cc:304] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.192973 2475 raft_consensus.cc:2747] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.199841 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3b5ccfb28464dcda7e964281f02f69a" } modify_external_catalogs: true
I20250625 07:52:39.200107 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b3b5ccfb28464dcda7e964281f02f69a" } modify_external_catalogs: true
W20250625 07:52:39.200152 850 consensus_peers.cc:487] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:39.200462 2475 raft_consensus.cc:491] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.200574 2475 raft_consensus.cc:513] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.201344 2475 leader_election.cc:290] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.202194 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.202684 850 leader_election.cc:343] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.203339 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:39.203861 851 leader_election.cc:343] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.203922 1158 tablet_bootstrap.cc:492] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.204015 851 leader_election.cc:304] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.204073 1158 ts_tablet_manager.cc:1397] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20250625 07:52:39.204443 2475 raft_consensus.cc:2747] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.204859 1158 raft_consensus.cc:357] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.205039 1158 raft_consensus.cc:383] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.205101 1158 raft_consensus.cc:738] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.205314 1158 consensus_queue.cc:260] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.207789 1160 ts_tablet_manager.cc:1918] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.209780 1158 ts_tablet_manager.cc:1428] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:39.210234 1158 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.212198 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bfcef152c9844a59989c532bd60c07d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.207906 1160 log.cc:1199] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/1c8694f09dcf47ab9f5fc8854c21c331
I20250625 07:52:39.212781 1160 ts_tablet_manager.cc:1939] T 1c8694f09dcf47ab9f5fc8854c21c331 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.213712 1160 tablet_replica.cc:331] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.213856 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 1c8694f09dcf47ab9f5fc8854c21c331 (table default.Table_5a458f32a2864c59a2c926d7760d2f13 [id=9e7bb49609f34ef4adadd800b22843e0]) successfully deleted
W20250625 07:52:39.215006 713 leader_election.cc:343] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.215490 2603 raft_consensus.cc:491] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.215605 2603 raft_consensus.cc:513] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.216140 2603 leader_election.cc:290] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.216835 2506 raft_consensus.cc:491] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.217090 1158 tablet_bootstrap.cc:654] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.217018 2506 raft_consensus.cc:513] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.217784 2506 leader_election.cc:290] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.219081 986 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 086c9321a70e44b5bb3ea419bb466fbe
W20250625 07:52:39.220538 985 leader_election.cc:343] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 086c9321a70e44b5bb3ea419bb466fbe
I20250625 07:52:39.220654 985 leader_election.cc:304] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.220906 2506 raft_consensus.cc:2747] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.225294 1160 raft_consensus.cc:2241] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.225518 1160 raft_consensus.cc:2270] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.225924 1162 tablet_bootstrap.cc:654] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.226082 1163 tablet_bootstrap.cc:492] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.226269 1163 ts_tablet_manager.cc:1397] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.008s
I20250625 07:52:39.227079 1163 raft_consensus.cc:357] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.227298 1163 raft_consensus.cc:383] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.227383 1163 raft_consensus.cc:738] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.227411 1160 ts_tablet_manager.cc:1905] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.227630 1163 consensus_queue.cc:260] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.228104 1163 ts_tablet_manager.cc:1428] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:39.228402 1163 tablet_bootstrap.cc:492] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.229970 1163 tablet_bootstrap.cc:654] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:39.231251 713 leader_election.cc:343] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.231431 713 leader_election.cc:304] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.231678 2603 raft_consensus.cc:2747] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:39.232180 713 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.232430 713 leader_election.cc:343] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.232985 713 consensus_peers.cc:487] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:39.233507 713 leader_election.cc:343] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.235313 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:39.236371 2506 raft_consensus.cc:491] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.236493 2506 raft_consensus.cc:513] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.237061 2506 leader_election.cc:290] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.238673 985 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 110fe299c7c44df69f459da72a1138a5
I20250625 07:52:39.234628 2534 tablet.cc:1722] T 2db37ece7a964f47911fe4fc7434cd07 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:39.239154 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_db6b195c7fa546a2a9c4532db1a2c058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.236141 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7ed91509ba94562a9207c7d7c0d3e2b" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:39.239579 914 raft_consensus.cc:3058] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:39.239725 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db6b195c7fa546a2a9c4532db1a2c058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.240463 1158 tablet_bootstrap.cc:492] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.240584 1158 ts_tablet_manager.cc:1397] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250625 07:52:39.241374 1158 raft_consensus.cc:357] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.241542 1158 raft_consensus.cc:383] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.241609 1158 raft_consensus.cc:738] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.241822 1158 consensus_queue.cc:260] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.242309 1158 ts_tablet_manager.cc:1428] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:39.242399 914 raft_consensus.cc:2466] T d7ed91509ba94562a9207c7d7c0d3e2b P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
W20250625 07:52:39.249761 986 leader_election.cc:343] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 110fe299c7c44df69f459da72a1138a5
I20250625 07:52:39.249917 986 leader_election.cc:304] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.250433 2506 raft_consensus.cc:2747] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.250936 1158 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.253556 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet cf175e17865e40ceb26aa955b91c2c13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.257319 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a737026476024d37af115083c4585882" } modify_external_catalogs: true
I20250625 07:52:39.257496 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a737026476024d37af115083c4585882" } modify_external_catalogs: true
I20250625 07:52:39.257980 1158 tablet_bootstrap.cc:654] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.259913 2407 ts_tablet_manager.cc:1918] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.260087 2407 log.cc:1199] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/c3d8b6b5c06248879e6052f6ae7fe46e
I20250625 07:52:39.260416 2407 ts_tablet_manager.cc:1939] T c3d8b6b5c06248879e6052f6ae7fe46e P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:39.261214 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1346) took 1983 ms. Trace:
I20250625 07:52:39.263386 1163 tablet_bootstrap.cc:492] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.263571 1163 ts_tablet_manager.cc:1397] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250625 07:52:39.264341 1163 raft_consensus.cc:357] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.264542 1163 raft_consensus.cc:383] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.264617 1163 raft_consensus.cc:738] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.264853 1163 consensus_queue.cc:260] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.265357 1163 ts_tablet_manager.cc:1428] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:39.265650 1163 tablet_bootstrap.cc:492] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.278550 1163 tablet_bootstrap.cc:654] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.278899 1158 tablet_bootstrap.cc:492] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.279063 1158 ts_tablet_manager.cc:1397] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250625 07:52:39.280136 1161 tablet_replica.cc:331] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:39.280143 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1536933d04014446ac8be0a896f0e6f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1536933d04014446ac8be0a896f0e6f3 already in progress: creating tablet
W20250625 07:52:39.280403 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ec1e58fe6d72451682bd47d7238560b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1e58fe6d72451682bd47d7238560b6 already in progress: creating tablet
W20250625 07:52:39.280608 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet dbdef039d3f842e5957c5bab85bedd42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbdef039d3f842e5957c5bab85bedd42 already in progress: creating tablet
I20250625 07:52:39.282682 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:39.282855 914 raft_consensus.cc:2466] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:39.286471 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.286794 1158 raft_consensus.cc:357] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.287010 1158 raft_consensus.cc:383] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.287109 1158 raft_consensus.cc:738] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.287514 1161 raft_consensus.cc:2241] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.287730 1158 consensus_queue.cc:260] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.288281 1158 ts_tablet_manager.cc:1428] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 07:52:39.288935 1158 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.290673 1158 tablet_bootstrap.cc:654] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.294891 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 03d999941e2a4265a64df7ec106a89a7
I20250625 07:52:39.287739 1161 raft_consensus.cc:2270] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.295701 1161 ts_tablet_manager.cc:1905] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.298841 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 03d999941e2a4265a64df7ec106a89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.300055 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 03d999941e2a4265a64df7ec106a89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.283418 2604 raft_consensus.cc:491] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.300885 2604 raft_consensus.cc:513] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.301522 2604 leader_election.cc:290] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:39.284309 713 consensus_peers.cc:487] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bcfe7c3c54741fbb0aca69650fc8eee. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:39.301895 713 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:39.302045 713 leader_election.cc:304] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.302284 2604 raft_consensus.cc:2747] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.303118 2407 rpcz_store.cc:276] 0625 07:52:37.277953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.277996 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:39.073670 (+1795674us) ts_tablet_manager.cc:1145] Deleting tablet c3d8b6b5c06248879e6052f6ae7fe46e
0625 07:52:39.073681 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.205720 (+132039us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.259801 (+ 54081us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.261195 (+ 1394us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795498}
I20250625 07:52:39.305444 1158 tablet_bootstrap.cc:492] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.305590 1158 ts_tablet_manager.cc:1397] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20250625 07:52:39.305634 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1347) took 2027 ms. Trace:
I20250625 07:52:39.305814 2407 rpcz_store.cc:276] 0625 07:52:37.278278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.278315 (+ 37us) service_pool.cc:225] Handling call
0625 07:52:39.305455 (+2027140us) ts_tablet_manager.cc:1145] Deleting tablet 0657163b9a0d4f7f99f57f7fd1f32462
0625 07:52:39.305467 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.305617 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2026984}
I20250625 07:52:39.306176 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1348) took 2027 ms. Trace:
I20250625 07:52:39.306305 2407 rpcz_store.cc:276] 0625 07:52:37.278560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.278595 (+ 35us) service_pool.cc:225] Handling call
0625 07:52:39.306104 (+2027509us) ts_tablet_manager.cc:1145] Deleting tablet 07a07af928ff4d3d894b074ef0cf44eb
0625 07:52:39.306112 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.306163 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027330}
I20250625 07:52:39.306384 1158 raft_consensus.cc:357] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.306568 1158 raft_consensus.cc:383] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.306656 1158 raft_consensus.cc:738] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.306862 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1349) took 2028 ms. Trace:
I20250625 07:52:39.306887 1158 consensus_queue.cc:260] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.307452 1158 ts_tablet_manager.cc:1428] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.307749 1158 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.309469 1158 tablet_bootstrap.cc:654] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.309325 2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3eb542ac24f6405687c1067f2885bec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a3faeffaa4422ea73b98ee74768a4a" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38477 } } peers { permanent_uuid: "789c06e7de2842a1b40e97e0778ab6c9" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 43317 } } peers { permanent_uuid: "3eb542ac24f6405687c1067f2885bec6" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 33457 } } }
I20250625 07:52:39.309540 2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 [sys.catalog]: This master's current role is: LEADER
I20250625 07:52:39.307011 2407 rpcz_store.cc:276] 0625 07:52:37.278871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.278905 (+ 34us) service_pool.cc:225] Handling call
0625 07:52:39.306692 (+2027787us) ts_tablet_manager.cc:1145] Deleting tablet 0b7daf759e324c1db853fdbaab76c721
0625 07:52:39.306701 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.306846 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027637}
I20250625 07:52:39.311317 2407 tablet_replica.cc:331] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:39.311540 2407 raft_consensus.cc:2241] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.311718 2407 raft_consensus.cc:2270] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.314702 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet fdad7dc1b27e4d87a747642a6d47230c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.315235 1160 ts_tablet_manager.cc:1918] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.319396 1158 tablet_bootstrap.cc:492] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.319574 1158 ts_tablet_manager.cc:1397] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20250625 07:52:39.320338 1158 raft_consensus.cc:357] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.320546 1158 raft_consensus.cc:383] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.320627 1158 raft_consensus.cc:738] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.320879 1158 consensus_queue.cc:260] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.321415 1158 ts_tablet_manager.cc:1428] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:39.321725 1158 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.323304 1158 tablet_bootstrap.cc:654] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.323889 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1bb389d88a7c49ea86fc830f2393cb07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.324548 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bb389d88a7c49ea86fc830f2393cb07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.305267 713 leader_election.cc:304] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.325062 2604 raft_consensus.cc:2802] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:39.325225 2604 raft_consensus.cc:491] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:39.325368 2604 raft_consensus.cc:3058] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.304563 1162 tablet_bootstrap.cc:492] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:39.325713 1162 ts_tablet_manager.cc:1397] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.152s user 0.003s sys 0.000s
I20250625 07:52:39.325976 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_330c8b7943384fccb11c9490fd7dd310"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.326217 1158 tablet_bootstrap.cc:492] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.326368 1158 ts_tablet_manager.cc:1397] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
W20250625 07:52:39.326361 713 leader_election.cc:343] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: fe630469fe7c4f4bac28277896eddc9c
W20250625 07:52:39.326517 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_330c8b7943384fccb11c9490fd7dd310 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.326565 713 leader_election.cc:304] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.326864 2603 raft_consensus.cc:2747] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.327163 1158 raft_consensus.cc:357] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.327361 1158 raft_consensus.cc:383] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.327457 1158 raft_consensus.cc:738] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.327657 1158 consensus_queue.cc:260] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.328111 1158 ts_tablet_manager.cc:1428] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.328399 1158 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.329950 1158 tablet_bootstrap.cc:654] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:39.332376 713 consensus_peers.cc:487] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abb4d9dc2da7453d8336d5589dda3a74. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:39.332994 1158 tablet_bootstrap.cc:492] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.332974 713 leader_election.cc:304] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:39.333155 1158 ts_tablet_manager.cc:1397] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250625 07:52:39.333364 2603 raft_consensus.cc:2802] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.333477 2603 raft_consensus.cc:695] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.334059 713 leader_election.cc:304] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:39.334334 2498 raft_consensus.cc:2802] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:39.334403 2498 raft_consensus.cc:491] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:39.334460 2498 raft_consensus.cc:3058] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.334517 1158 raft_consensus.cc:357] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.334707 1158 raft_consensus.cc:383] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.334780 1158 raft_consensus.cc:738] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.334785 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0fe117874dda4408934ce4e1687c10d6
I20250625 07:52:39.335007 1158 consensus_queue.cc:260] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.335510 1158 ts_tablet_manager.cc:1428] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:39.335806 1158 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.337585 1158 tablet_bootstrap.cc:654] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.337610 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.338099 2494 raft_consensus.cc:491] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.338230 2494 raft_consensus.cc:513] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.338321 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet c3d8b6b5c06248879e6052f6ae7fe46e (table default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]) successfully deleted
W20250625 07:52:39.338558 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0657163b9a0d4f7f99f57f7fd1f32462 because the tablet was not found. No further retry: Not found: Tablet not found: 0657163b9a0d4f7f99f57f7fd1f32462
W20250625 07:52:39.338725 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 07a07af928ff4d3d894b074ef0cf44eb because the tablet was not found. No further retry: Not found: Tablet not found: 07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:39.338721 2494 leader_election.cc:290] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.340378 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fe117874dda4408934ce4e1687c10d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.347275 1158 tablet_bootstrap.cc:492] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.347443 1158 ts_tablet_manager.cc:1397] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20250625 07:52:39.348170 1158 raft_consensus.cc:357] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.348366 1158 raft_consensus.cc:383] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.348448 1158 raft_consensus.cc:738] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.348649 1158 consensus_queue.cc:260] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.348798 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2fe7fe88ebf948fdaa4f5bc20b6f9378" } new_table_name: "default.Table_2ef7f118a90d46c3a408c4609e30cb57" modify_external_catalogs: true
I20250625 07:52:39.349157 1158 ts_tablet_manager.cc:1428] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:39.349459 1158 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.350176 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fffd92726b724fd09763acbb604f0941" } new_table_name: "default.Table_5c37d7ee9cf14b9b80231a5991ce9acf" modify_external_catalogs: true
I20250625 07:52:39.351442 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7d8f8cc6cbff4a34b036a097f4fa40ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.351787 2407 ts_tablet_manager.cc:1905] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:39.352016 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d8f8cc6cbff4a34b036a097f4fa40ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.357508 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 03d999941e2a4265a64df7ec106a89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.359792 2498 raft_consensus.cc:513] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.360366 2498 leader_election.cc:290] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.360533 2604 raft_consensus.cc:513] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.360854 2603 consensus_queue.cc:237] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.361027 2604 leader_election.cc:290] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.362916 1158 tablet_bootstrap.cc:654] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:39.363337 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4fbad23ebab481cb2ce4dd830e25718 consensus state has no leader
I20250625 07:52:39.363855 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d5fe1be63e9842068b71adac8fc011d4" } new_table_name: "default.Table_edd9a0a8a5ec47e6bdfb9264fdbc961d" modify_external_catalogs: true
I20250625 07:52:39.367606 1162 raft_consensus.cc:357] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.367861 1162 raft_consensus.cc:383] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.367980 1162 raft_consensus.cc:738] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.368182 1162 consensus_queue.cc:260] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.368757 1162 ts_tablet_manager.cc:1428] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250625 07:52:39.369184 1162 tablet_bootstrap.cc:492] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:39.370946 1162 tablet_bootstrap.cc:654] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.371824 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_114f4c0ed74b446b9ddf211e56589e6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.372510 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_114f4c0ed74b446b9ddf211e56589e6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.373893 1158 tablet_bootstrap.cc:492] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.374107 1158 ts_tablet_manager.cc:1397] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20250625 07:52:39.375020 1158 raft_consensus.cc:357] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.375218 1158 raft_consensus.cc:383] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.375308 1158 raft_consensus.cc:738] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.375551 1158 consensus_queue.cc:260] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.376077 1158 ts_tablet_manager.cc:1428] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:39.376420 1158 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.376947 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 4dbf323dfa034cc78904cb73077f4ccf from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:39.378089 1158 tablet_bootstrap.cc:654] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.380563 1158 tablet_bootstrap.cc:492] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.380726 1158 ts_tablet_manager.cc:1397] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250625 07:52:39.381340 1162 tablet_bootstrap.cc:492] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:39.381503 1162 ts_tablet_manager.cc:1397] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250625 07:52:39.381467 1158 raft_consensus.cc:357] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.381644 1158 raft_consensus.cc:383] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.381737 1158 raft_consensus.cc:738] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.381956 1158 consensus_queue.cc:260] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.382468 1158 ts_tablet_manager.cc:1428] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.382761 1158 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.384761 1158 tablet_bootstrap.cc:654] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.382601 1162 raft_consensus.cc:357] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.385200 1162 raft_consensus.cc:383] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.385304 1162 raft_consensus.cc:738] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.385553 1162 consensus_queue.cc:260] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.387235 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fb211d352ae4ee6b433376787420700 has no consensus state
I20250625 07:52:39.388165 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_afb2c447878f4e4ea10bbe04bcdaa8e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.389025 1162 ts_tablet_manager.cc:1428] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:39.389384 1162 tablet_bootstrap.cc:492] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:39.318110 2475 raft_consensus.cc:491] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.392417 2475 raft_consensus.cc:513] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.393270 2475 leader_election.cc:290] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.393425 2475 raft_consensus.cc:491] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.393530 2475 raft_consensus.cc:513] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.394021 2475 leader_election.cc:290] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.335791 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:39.339694 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fe117874dda4408934ce4e1687c10d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.394295 914 raft_consensus.cc:2466] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:39.394523 1158 tablet_bootstrap.cc:492] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.394704 1158 ts_tablet_manager.cc:1397] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250625 07:52:39.395162 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8ebaaa067c204fac84a2d6a6de6dbcd8" } new_table_name: "default.Table_1e81a97a712e4ba0aa4f1a63186851db" modify_external_catalogs: true
I20250625 07:52:39.395231 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f7240577a874a59b3268ff7f0c1beec" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.395529 1158 raft_consensus.cc:357] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.395637 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90f2bef7baa54f1a96b3b59921d59457" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.395718 1158 raft_consensus.cc:383] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.395792 1158 raft_consensus.cc:738] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.396041 1158 consensus_queue.cc:260] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.396229 851 leader_election.cc:343] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.396401 851 leader_election.cc:343] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.396560 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afb2c447878f4e4ea10bbe04bcdaa8e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.396577 1158 ts_tablet_manager.cc:1428] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.396872 1158 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.397411 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f7240577a874a59b3268ff7f0c1beec" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.398231 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90f2bef7baa54f1a96b3b59921d59457" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.398636 1158 tablet_bootstrap.cc:654] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:39.398720 850 leader_election.cc:343] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.398809 850 leader_election.cc:304] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:39.399180 850 leader_election.cc:343] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.399271 850 leader_election.cc:304] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.399811 2475 raft_consensus.cc:2747] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.400000 2475 raft_consensus.cc:2747] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.401151 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_08fde6e3798a4aa0ab2006ff6532dd63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.402454 1162 tablet_bootstrap.cc:654] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.406917 1158 tablet_bootstrap.cc:492] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.407110 1158 ts_tablet_manager.cc:1397] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20250625 07:52:39.315390 1160 log.cc:1199] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e181a3f1ae2c49349718d842c166b1e6
I20250625 07:52:39.407913 1158 raft_consensus.cc:357] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.408115 1158 raft_consensus.cc:383] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.408218 1158 raft_consensus.cc:738] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.408393 1160 ts_tablet_manager.cc:1939] T e181a3f1ae2c49349718d842c166b1e6 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.408470 1158 consensus_queue.cc:260] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.409029 1158 ts_tablet_manager.cc:1428] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:39.409341 1158 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.411087 1158 tablet_bootstrap.cc:654] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:39.411728 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08fde6e3798a4aa0ab2006ff6532dd63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.412842 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fe117874dda4408934ce4e1687c10d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:39.409811 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 448a7d67fc014411ab294b19c65637ee consensus state has no leader
W20250625 07:52:39.413313 713 leader_election.cc:343] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.413725 1158 tablet_bootstrap.cc:492] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.413872 1158 ts_tablet_manager.cc:1397] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250625 07:52:39.414553 1158 raft_consensus.cc:357] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.414741 1158 raft_consensus.cc:383] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.414812 1158 raft_consensus.cc:738] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:39.415097 713 leader_election.cc:343] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.415062 1158 consensus_queue.cc:260] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.415560 1158 ts_tablet_manager.cc:1428] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.415860 1158 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.417621 1158 tablet_bootstrap.cc:654] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.417791 1160 tablet_replica.cc:331] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.418030 1160 raft_consensus.cc:2241] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.418190 1160 raft_consensus.cc:2270] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.418304 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e181a3f1ae2c49349718d842c166b1e6 (table default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]) successfully deleted
I20250625 07:52:39.418789 1160 ts_tablet_manager.cc:1905] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.421607 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_0d9011da7226470ab7282dcdbc637180 [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 2 got 0
I20250625 07:52:39.421837 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:39.422236 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
W20250625 07:52:39.423280 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61e076870efc4744b1224ddcb0eef3fa consensus state has no leader
I20250625 07:52:39.423780 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8fdc3220d53142faae41fd1f1500de56
I20250625 07:52:39.430891 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdc3220d53142faae41fd1f1500de56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.432370 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_87be1efdccf045a9877e5f62907de1f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.433090 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87be1efdccf045a9877e5f62907de1f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.434870 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5bcfcafad68c45a28e30214ffb855371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.435467 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bcfcafad68c45a28e30214ffb855371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.437031 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a59eca497c7e423b801c1f7e4e1ed131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.437664 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a59eca497c7e423b801c1f7e4e1ed131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.438169 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdc3220d53142faae41fd1f1500de56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.438967 1158 tablet_bootstrap.cc:492] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.439131 1158 ts_tablet_manager.cc:1397] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20250625 07:52:39.440001 1158 raft_consensus.cc:357] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.440232 1158 raft_consensus.cc:383] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.440356 1158 raft_consensus.cc:738] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.440541 1161 ts_tablet_manager.cc:1918] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.440789 1161 log.cc:1199] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/f04016cda7f746d49e30be3b1c9c18b6
I20250625 07:52:39.441259 1161 ts_tablet_manager.cc:1939] T f04016cda7f746d49e30be3b1c9c18b6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:39.442261 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
W20250625 07:52:39.445060 713 leader_election.cc:343] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.440604 1158 consensus_queue.cc:260] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.447124 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1973678168a84875b747897f08dd1bad" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.447340 1158 ts_tablet_manager.cc:1428] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250625 07:52:39.451300 2506 raft_consensus.cc:491] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.451457 2506 raft_consensus.cc:513] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.452062 2506 leader_election.cc:290] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.453133 2407 ts_tablet_manager.cc:1918] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:39.453171 986 leader_election.cc:343] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:39.454594 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:39.454676 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.454774 914 raft_consensus.cc:3058] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.455132 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 26db0effa17b4cacb1450a01f9631ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.457530 914 raft_consensus.cc:2466] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:39.461464 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:39.461642 911 raft_consensus.cc:3058] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.464147 911 raft_consensus.cc:2466] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:39.465590 2628 raft_consensus.cc:491] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.465786 2628 raft_consensus.cc:513] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.467156 2628 leader_election.cc:290] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.467559 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26db0effa17b4cacb1450a01f9631ddf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.467978 850 leader_election.cc:343] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.470904 2506 raft_consensus.cc:491] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.471140 1158 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:39.471174 985 leader_election.cc:343] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:39.471138 2506 raft_consensus.cc:513] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.472425 2506 leader_election.cc:290] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.450968 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_901005941b484ae7a758af04aa665f16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.463234 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 35f819ef1fa74982a50d6714de305535
I20250625 07:52:39.473049 1158 tablet_bootstrap.cc:654] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.473080 1161 tablet_replica.cc:331] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:39.473338 1161 raft_consensus.cc:2241] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.473534 1161 raft_consensus.cc:2270] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:39.473760 986 leader_election.cc:343] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0aeb7077591f466eb77e3096f6b1d74b
I20250625 07:52:39.474073 1161 ts_tablet_manager.cc:1905] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.474071 985 leader_election.cc:304] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.474648 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 35f819ef1fa74982a50d6714de305535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.474769 2506 raft_consensus.cc:2747] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:39.475521 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_901005941b484ae7a758af04aa665f16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.476516 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0352225d645c408ba8c74273ee5f56e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:39.477062 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61e076870efc4744b1224ddcb0eef3fa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:39.477766 1163 tablet_bootstrap.cc:492] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.477917 1163 ts_tablet_manager.cc:1397] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.212s user 0.018s sys 0.019s
I20250625 07:52:39.478729 1163 raft_consensus.cc:357] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.478905 1163 raft_consensus.cc:383] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.478967 1163 raft_consensus.cc:738] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.479167 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet f04016cda7f746d49e30be3b1c9c18b6 (table default.Table_634beccfc1a34257bd0bc04d1d020530 [id=989d9b62801c467e8bba2e03c0f13395]) successfully deleted
I20250625 07:52:39.479171 1163 consensus_queue.cc:260] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.479709 1163 ts_tablet_manager.cc:1428] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.479713 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 35f819ef1fa74982a50d6714de305535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.480020 1163 tablet_bootstrap.cc:492] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.480823 2407 log.cc:1199] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/a41ffd566593499299aba5513ccb2f30
I20250625 07:52:39.481173 2407 ts_tablet_manager.cc:1939] T a41ffd566593499299aba5513ccb2f30 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:39.481997 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1350) took 2202 ms. Trace:
I20250625 07:52:39.482599 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 62aaff3c000e4e69b52222aa47260c3a (table default.Table_e01d3c58925248249dc9bff2b7be111c [id=79d68081c3a643109237f5d5478df82c]). Expected version 1 got 0
I20250625 07:52:39.482709 1163 tablet_bootstrap.cc:654] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.482834 589 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:39.486259 2475 consensus_queue.cc:1035] T 1796a3305cb94ccb9a0596643a736a96 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:39.489307 2407 rpcz_store.cc:276] 0625 07:52:37.279147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.279188 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:39.311272 (+2032084us) ts_tablet_manager.cc:1145] Deleting tablet a41ffd566593499299aba5513ccb2f30
0625 07:52:39.311281 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.449475 (+138194us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.453015 (+ 3540us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.481973 (+ 28958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2031908}
W20250625 07:52:39.489763 985 leader_election.cc:343] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0aeb7077591f466eb77e3096f6b1d74b
I20250625 07:52:39.489843 2407 tablet_replica.cc:331] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:39.489918 985 leader_election.cc:304] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.490202 2407 raft_consensus.cc:2241] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.490401 2407 raft_consensus.cc:2270] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.490931 2407 ts_tablet_manager.cc:1905] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.475669 1162 tablet_bootstrap.cc:492] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:39.491972 1162 ts_tablet_manager.cc:1397] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.005s
I20250625 07:52:39.492821 1162 raft_consensus.cc:357] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.466833 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26db0effa17b4cacb1450a01f9631ddf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.461464 713 leader_election.cc:304] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.493345 2498 raft_consensus.cc:2802] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:39.493419 2498 raft_consensus.cc:491] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:39.493484 2498 raft_consensus.cc:3058] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:39.493573 851 leader_election.cc:343] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.493685 851 leader_election.cc:304] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.494067 2475 raft_consensus.cc:2747] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.496282 2498 raft_consensus.cc:513] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.496310 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb211d352ae4ee6b433376787420700 (DEFAULT_TABLE table=default.Table_edd9a0a8a5ec47e6bdfb9264fdbc961d [id=97abf3cac6584696ba07c6db0822b5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.496742 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb211d352ae4ee6b433376787420700. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.498149 713 leader_election.cc:343] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1973678168a84875b747897f08dd1bad
I20250625 07:52:39.499604 713 leader_election.cc:304] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:39.499897 2604 raft_consensus.cc:2802] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.500018 2604 raft_consensus.cc:695] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.500258 2604 consensus_queue.cc:237] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.501408 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9185bc88074a4d69beb1f796eac0428c (DEFAULT_TABLE table=default.Table_08987651f24242e4a8e9eb2e4223006c [id=d130fbd910e448bd8be29da5cc849e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.501829 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9185bc88074a4d69beb1f796eac0428c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.502261 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9a780105a1f44d1f8ed934b8a1ee6b32" } modify_external_catalogs: true
I20250625 07:52:39.502440 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9a780105a1f44d1f8ed934b8a1ee6b32" } modify_external_catalogs: true
I20250625 07:52:39.502671 2498 leader_election.cc:290] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.504184 713 leader_election.cc:304] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:39.504599 2498 raft_consensus.cc:2802] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.505584 883 tablet_service.cc:1468] Processing CreateTablet for tablet a0cda2f874784ab2a53f7b12fce87a6b (DEFAULT_TABLE table=default.Table_fc8419bb058a41ccb3c725d395cd32ac [id=1d69100b67314c6c9c5d7a6e296c72cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.505955 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cda2f874784ab2a53f7b12fce87a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.508013 878 tablet_service.cc:1468] Processing CreateTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.508458 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9363b569fb84cc0bf55e0c6e2dbc608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.509014 885 tablet_service.cc:1468] Processing CreateTablet for tablet b6330db97ed84e6485d405c75e92ead4 (DEFAULT_TABLE table=default.Table_7e91ee1c67a4449bb67de5b4a113d8f8 [id=651f7c80074642e09b12d7a125402218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.509393 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6330db97ed84e6485d405c75e92ead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.513595 2498 raft_consensus.cc:695] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.514006 2498 consensus_queue.cc:237] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.514631 2603 raft_consensus.cc:491] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.514748 2603 raft_consensus.cc:513] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.515254 2603 leader_election.cc:290] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.521332 2506 raft_consensus.cc:2747] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.521725 2506 raft_consensus.cc:491] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.521852 2506 raft_consensus.cc:513] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.521941 883 tablet_service.cc:1468] Processing CreateTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.522323 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c7ac8ff934a8ca40ac480c7c352ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.522500 2506 leader_election.cc:290] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.523728 986 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 7db12504a1a54add920561dae27d2d2b
I20250625 07:52:39.527240 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0aeb7077591f466eb77e3096f6b1d74b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.528443 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 07a07af928ff4d3d894b074ef0cf44eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.530668 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0657163b9a0d4f7f99f57f7fd1f32462 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.533175 885 tablet_service.cc:1468] Processing CreateTablet for tablet 52633d8b1d6e48b08ff52263c8a19496 (DEFAULT_TABLE table=default.Table_65a20180326d44a099e0216cbff039fd [id=d1af1dd3406a4a5e831139910b6e63ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.533687 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52633d8b1d6e48b08ff52263c8a19496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.536077 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4dbf323dfa034cc78904cb73077f4ccf
I20250625 07:52:39.536123 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7daf759e324c1db853fdbaab76c721 with delete_type TABLET_DATA_DELETED (replaced by tablet 03d999941e2a4265a64df7ec106a89a7) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.536470 591 catalog_manager.cc:6652] ReplaceTablet: tablet 4dbf323dfa034cc78904cb73077f4ccf deleted and replaced by tablet e92dc108e9fb4942a51cec78da1f0930
I20250625 07:52:39.537009 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fdc3220d53142faae41fd1f1500de56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.537434 750 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.537971 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet c263b671be384cd097041c60306034de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:16 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.538096 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf with delete_type TABLET_DATA_DELETED (replaced by tablet e92dc108e9fb4942a51cec78da1f0930) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.538488 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet c0cc3ba09d5e4c3d88e74fffc97eea8b with delete_type TABLET_DATA_DELETED (replaced by tablet d74924e8dee943e0ae4c93f0db1d75f4) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.538523 886 tablet_service.cc:1468] Processing CreateTablet for tablet f2fdc74b19bd4c559a497c7388a2b9a8 (DEFAULT_TABLE table=default.Table_97d80d02416046f1b32d44edcd8ab622 [id=376534962f26464590b71eb8637e4c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.538919 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fdc74b19bd4c559a497c7388a2b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.540184 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb211d352ae4ee6b433376787420700 (DEFAULT_TABLE table=default.Table_edd9a0a8a5ec47e6bdfb9264fdbc961d [id=97abf3cac6584696ba07c6db0822b5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.540592 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb211d352ae4ee6b433376787420700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.541008 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1973678168a84875b747897f08dd1bad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.544135 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f51ef840a3ed40bd939b3fe62df3cda2
I20250625 07:52:39.546253 2603 raft_consensus.cc:491] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.546375 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.546433 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19] alter complete (version 3)
I20250625 07:52:39.546394 2603 raft_consensus.cc:513] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.546732 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.546994 2603 leader_election.cc:290] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.547039 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 96a1f823fd3b47c9b6d1cf5647f9a68a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.547303 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62dfb8f20f4caeaa79ea1e85d99930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.549127 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aef7709d3ff64d91a06278b1080f77ba" } modify_external_catalogs: true
I20250625 07:52:39.549343 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aef7709d3ff64d91a06278b1080f77ba" } modify_external_catalogs: true
I20250625 07:52:39.549918 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9185bc88074a4d69beb1f796eac0428c (DEFAULT_TABLE table=default.Table_08987651f24242e4a8e9eb2e4223006c [id=d130fbd910e448bd8be29da5cc849e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.550412 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9185bc88074a4d69beb1f796eac0428c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.550812 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cc3b3771179f4d57b2d2a7e965e87550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.551342 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc3b3771179f4d57b2d2a7e965e87550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.552934 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4970ea915b7042629008e088f9d45013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.553507 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4970ea915b7042629008e088f9d45013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.556885 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 35f819ef1fa74982a50d6714de305535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.556891 879 tablet_service.cc:1515] Processing DeleteTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.557515 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4fb211d352ae4ee6b433376787420700 (DEFAULT_TABLE table=default.Table_edd9a0a8a5ec47e6bdfb9264fdbc961d [id=97abf3cac6584696ba07c6db0822b5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.557899 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fb211d352ae4ee6b433376787420700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.559579 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4fca218ee8594edeb0eaed16c81890dd (DEFAULT_TABLE table=default.Table_9a51b748961b4e90971558af21f7b836 [id=0f70d5c1f1a145699c7248e90258fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.560057 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fca218ee8594edeb0eaed16c81890dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.560981 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a0cda2f874784ab2a53f7b12fce87a6b (DEFAULT_TABLE table=default.Table_fc8419bb058a41ccb3c725d395cd32ac [id=1d69100b67314c6c9c5d7a6e296c72cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.561401 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cda2f874784ab2a53f7b12fce87a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.562884 1012 tablet_service.cc:1468] Processing CreateTablet for tablet b6330db97ed84e6485d405c75e92ead4 (DEFAULT_TABLE table=default.Table_7e91ee1c67a4449bb67de5b4a113d8f8 [id=651f7c80074642e09b12d7a125402218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.563236 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6330db97ed84e6485d405c75e92ead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.570003 2603 raft_consensus.cc:491] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.570155 2603 raft_consensus.cc:513] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.570717 2603 leader_election.cc:290] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.571063 742 tablet_service.cc:1468] Processing CreateTablet for tablet a0cda2f874784ab2a53f7b12fce87a6b (DEFAULT_TABLE table=default.Table_fc8419bb058a41ccb3c725d395cd32ac [id=1d69100b67314c6c9c5d7a6e296c72cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.571434 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cda2f874784ab2a53f7b12fce87a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.572646 1023 tablet_service.cc:1468] Processing CreateTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.573042 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c7ac8ff934a8ca40ac480c7c352ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.575018 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_82cd9b0cdb87402c90df8c979e56570a" } new_table_name: "default.Table_cb77d581902c4391ac8bcdca79163105" modify_external_catalogs: true
I20250625 07:52:39.576813 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_84bea214164245c8a5debd8bd9d38fb7" } new_table_name: "default.Table_3eddf8bcebcc4c6b8c9545d48089145a" modify_external_catalogs: true
I20250625 07:52:39.579524 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_880f1c0b45ed4ab38c040de1a705f41c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.580173 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_880f1c0b45ed4ab38c040de1a705f41c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.580572 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 52633d8b1d6e48b08ff52263c8a19496 (DEFAULT_TABLE table=default.Table_65a20180326d44a099e0216cbff039fd [id=d1af1dd3406a4a5e831139910b6e63ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.581048 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52633d8b1d6e48b08ff52263c8a19496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.580991 758 tablet_service.cc:1468] Processing CreateTablet for tablet 9185bc88074a4d69beb1f796eac0428c (DEFAULT_TABLE table=default.Table_08987651f24242e4a8e9eb2e4223006c [id=d130fbd910e448bd8be29da5cc849e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.581786 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9185bc88074a4d69beb1f796eac0428c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.583539 878 tablet_service.cc:1468] Processing CreateTablet for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4 (DEFAULT_TABLE table=default.Table_01e01fa76e6e4c7fac6a613140fab849 [id=448bed3175eb43828186af1a5c218123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.583909 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.587487 1162 raft_consensus.cc:383] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.587599 1162 raft_consensus.cc:738] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.587813 1162 consensus_queue.cc:260] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.588536 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.588970 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9363b569fb84cc0bf55e0c6e2dbc608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.588951 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_a963a1a41d5a4ac8bc0797be58a18631" } new_table_name: "default.Table_57f90385582f4bb38f2e7dd0287d924b" modify_external_catalogs: true
I20250625 07:52:39.589916 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f2fdc74b19bd4c559a497c7388a2b9a8 (DEFAULT_TABLE table=default.Table_97d80d02416046f1b32d44edcd8ab622 [id=376534962f26464590b71eb8637e4c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.590459 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fdc74b19bd4c559a497c7388a2b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.590875 746 tablet_service.cc:1468] Processing CreateTablet for tablet b6330db97ed84e6485d405c75e92ead4 (DEFAULT_TABLE table=default.Table_7e91ee1c67a4449bb67de5b4a113d8f8 [id=651f7c80074642e09b12d7a125402218]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.591238 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6330db97ed84e6485d405c75e92ead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.596403 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c122910d938045b480a31da589781477"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.597129 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c122910d938045b480a31da589781477 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.598217 756 tablet_service.cc:1468] Processing CreateTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.598574 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15c7ac8ff934a8ca40ac480c7c352ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.602573 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e4b87309b76f4f0e9dde120d9f755c44
I20250625 07:52:39.604708 756 tablet_service.cc:1468] Processing CreateTablet for tablet 52633d8b1d6e48b08ff52263c8a19496 (DEFAULT_TABLE table=default.Table_65a20180326d44a099e0216cbff039fd [id=d1af1dd3406a4a5e831139910b6e63ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.605072 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52633d8b1d6e48b08ff52263c8a19496. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.605155 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.609128 746 tablet_service.cc:1468] Processing CreateTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.609504 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9363b569fb84cc0bf55e0c6e2dbc608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.592590 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf with delete_type TABLET_DATA_DELETED (replaced by tablet e92dc108e9fb4942a51cec78da1f0930) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.597862 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.598205 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 4fca218ee8594edeb0eaed16c81890dd (DEFAULT_TABLE table=default.Table_9a51b748961b4e90971558af21f7b836 [id=0f70d5c1f1a145699c7248e90258fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.598562 2506 raft_consensus.cc:491] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.599285 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4 (DEFAULT_TABLE table=default.Table_01e01fa76e6e4c7fac6a613140fab849 [id=448bed3175eb43828186af1a5c218123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.612707 2506 raft_consensus.cc:513] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.612900 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.613430 2506 leader_election.cc:290] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.613679 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fca218ee8594edeb0eaed16c81890dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.613740 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abb4d9dc2da7453d8336d5589dda3a74" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.614323 889 tablet_service.cc:1468] Processing CreateTablet for tablet ad95e981ba494cb3bdbc5e6180ff9826 (DEFAULT_TABLE table=default.Table_a6d1fa25a05b4d42aa04d2a632d23bdc [id=f753c97b91ac403999f698c37e0f5ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.614675 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad95e981ba494cb3bdbc5e6180ff9826. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.599561 985 leader_election.cc:343] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 7db12504a1a54add920561dae27d2d2b
I20250625 07:52:39.618724 985 leader_election.cc:304] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.619020 2506 raft_consensus.cc:2747] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.619428 589 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba] alter complete (version 1)
I20250625 07:52:39.619946 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.620270 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 7db12504a1a54add920561dae27d2d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.620334 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ad95e981ba494cb3bdbc5e6180ff9826 (DEFAULT_TABLE table=default.Table_a6d1fa25a05b4d42aa04d2a632d23bdc [id=f753c97b91ac403999f698c37e0f5ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.620523 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b9eb3bcbece34f14a05dc7d95ca7ece2
I20250625 07:52:39.620743 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad95e981ba494cb3bdbc5e6180ff9826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.621989 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b8146a0332fc4ef897cbf4a745c83b5f" } new_table_name: "default.Table_fb65979f3efd4371a7512d96f5f3640a" modify_external_catalogs: true
I20250625 07:52:39.623854 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f31104720614b028e373ae1500c31cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.624373 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f7240577a874a59b3268ff7f0c1beec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.626405 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_90b0fb7fffff488ca122029f4343c687" } new_table_name: "default.Table_03c8e125773b46619e7693fa2620d64f" modify_external_catalogs: true
W20250625 07:52:39.626648 985 leader_election.cc:343] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: abb4d9dc2da7453d8336d5589dda3a74
I20250625 07:52:39.626789 985 leader_election.cc:304] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.627247 2506 raft_consensus.cc:2747] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.628118 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bdc29c3cb4b44fdabdaa413f46834dcb" } modify_external_catalogs: true
I20250625 07:52:39.628306 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bdc29c3cb4b44fdabdaa413f46834dcb" } modify_external_catalogs: true
I20250625 07:52:39.628609 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.629451 1162 ts_tablet_manager.cc:1428] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.137s user 0.002s sys 0.000s
I20250625 07:52:39.630067 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.630587 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa with delete_type TABLET_DATA_DELETED (replaced by tablet fed43adfae594e3b91ed1012838c3317) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.630838 1162 tablet_bootstrap.cc:492] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:39.632370 1162 tablet_bootstrap.cc:654] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.634821 883 tablet_service.cc:1468] Processing CreateTablet for tablet 09cb7d5885654b5b920af2ab442ea59b (DEFAULT_TABLE table=default.Table_0c7f2c95095e4b52b0002c42225bf3a5 [id=8292af7d077b4623817377c8421f25c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.635466 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09cb7d5885654b5b920af2ab442ea59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.637336 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8317ed76301c4e6099172883e30b685a (DEFAULT_TABLE table=default.Table_887824db43f7441e822ae633b51194bd [id=53e10793ef44444188ac72fee6997cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.640142 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317ed76301c4e6099172883e30b685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.641616 755 tablet_service.cc:1468] Processing CreateTablet for tablet f2fdc74b19bd4c559a497c7388a2b9a8 (DEFAULT_TABLE table=default.Table_97d80d02416046f1b32d44edcd8ab622 [id=376534962f26464590b71eb8637e4c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.642001 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2fdc74b19bd4c559a497c7388a2b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.648638 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbf323dfa034cc78904cb73077f4ccf with delete_type TABLET_DATA_DELETED (replaced by tablet e92dc108e9fb4942a51cec78da1f0930) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.649905 1160 ts_tablet_manager.cc:1918] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.650115 1160 log.cc:1199] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/3488f79d09054f6dbfc6b54c1dee0d6c
I20250625 07:52:39.650545 1160 ts_tablet_manager.cc:1939] T 3488f79d09054f6dbfc6b54c1dee0d6c P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.651240 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 22411725aa5741c497328ce047c44d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.652074 1160 tablet_replica.cc:331] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:39.652094 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 3488f79d09054f6dbfc6b54c1dee0d6c (table default.Table_c121aa9cbde14c6bb4a4bcb574484208 [id=b6a77883382d4a6f8ff92139c24ba3be]) successfully deleted
I20250625 07:52:39.652330 1160 raft_consensus.cc:2241] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.652530 1160 raft_consensus.cc:2270] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.652575 2506 raft_consensus.cc:491] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:39.652561 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 60c01f0a75674c469b41fac638799afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c01f0a75674c469b41fac638799afa already in progress: creating tablet
I20250625 07:52:39.652755 2506 raft_consensus.cc:513] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:39.652887 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 03d999941e2a4265a64df7ec106a89a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d999941e2a4265a64df7ec106a89a7 already in progress: creating tablet
W20250625 07:52:39.653087 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 0fe117874dda4408934ce4e1687c10d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe117874dda4408934ce4e1687c10d6 already in progress: creating tablet
I20250625 07:52:39.653381 2506 leader_election.cc:290] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.654844 986 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 6e5a207d515449e298936d8bb532437a
I20250625 07:52:39.657343 739 tablet_service.cc:1468] Processing CreateTablet for tablet 4fca218ee8594edeb0eaed16c81890dd (DEFAULT_TABLE table=default.Table_9a51b748961b4e90971558af21f7b836 [id=0f70d5c1f1a145699c7248e90258fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.657788 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fca218ee8594edeb0eaed16c81890dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.657610 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 8317ed76301c4e6099172883e30b685a (DEFAULT_TABLE table=default.Table_887824db43f7441e822ae633b51194bd [id=53e10793ef44444188ac72fee6997cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.658178 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317ed76301c4e6099172883e30b685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.663481 1158 tablet_bootstrap.cc:492] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:39.663667 1158 ts_tablet_manager.cc:1397] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.200s user 0.025s sys 0.024s
I20250625 07:52:39.664422 1158 raft_consensus.cc:357] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.664624 1158 raft_consensus.cc:383] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.664712 1158 raft_consensus.cc:738] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.664742 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.664944 1158 consensus_queue.cc:260] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.665380 755 tablet_service.cc:1468] Processing CreateTablet for tablet ad95e981ba494cb3bdbc5e6180ff9826 (DEFAULT_TABLE table=default.Table_a6d1fa25a05b4d42aa04d2a632d23bdc [id=f753c97b91ac403999f698c37e0f5ea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.665675 1158 ts_tablet_manager.cc:1428] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:39.665824 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad95e981ba494cb3bdbc5e6180ff9826. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.665782 985 leader_election.cc:343] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 6e5a207d515449e298936d8bb532437a
I20250625 07:52:39.665916 985 leader_election.cc:304] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.666258 2506 raft_consensus.cc:2747] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.667094 879 tablet_service.cc:1468] Processing CreateTablet for tablet 52af63b3b42a44ed831f202ff2601073 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.667668 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_22a7b61bc1664cb8b3634346932badb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.668391 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22a7b61bc1664cb8b3634346932badb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.668912 739 tablet_service.cc:1468] Processing CreateTablet for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4 (DEFAULT_TABLE table=default.Table_01e01fa76e6e4c7fac6a613140fab849 [id=448bed3175eb43828186af1a5c218123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.668915 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 09cb7d5885654b5b920af2ab442ea59b (DEFAULT_TABLE table=default.Table_0c7f2c95095e4b52b0002c42225bf3a5 [id=8292af7d077b4623817377c8421f25c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.669345 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a69104e7f8243cf9b51c4ed4ac7cdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.669881 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52af63b3b42a44ed831f202ff2601073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.673655 889 tablet_service.cc:1468] Processing CreateTablet for tablet da3d35622f5845baa3976c5505a00cab (DEFAULT_TABLE table=default.Table_313d5eb00cc14cc882bb2839b715189e [id=e37d990099c14d9da7da93e93a4fe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.674101 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3d35622f5845baa3976c5505a00cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.680363 752 tablet_service.cc:1515] Processing DeleteTablet for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.681049 752 tablet_service.cc:1468] Processing CreateTablet for tablet 09cb7d5885654b5b920af2ab442ea59b (DEFAULT_TABLE table=default.Table_0c7f2c95095e4b52b0002c42225bf3a5 [id=8292af7d077b4623817377c8421f25c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.681420 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09cb7d5885654b5b920af2ab442ea59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.684497 1163 tablet_bootstrap.cc:492] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.684617 1163 ts_tablet_manager.cc:1397] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.205s user 0.027s sys 0.027s
I20250625 07:52:39.684957 2628 raft_consensus.cc:491] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.685082 2628 raft_consensus.cc:513] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.685408 1163 raft_consensus.cc:357] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.685557 1163 raft_consensus.cc:383] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.685611 1163 raft_consensus.cc:738] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.685668 2628 leader_election.cc:290] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.686082 1163 consensus_queue.cc:260] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.686367 893 tablet_service.cc:1468] Processing CreateTablet for tablet 68ced4b8365e4fd79c2c5934eb570e7b (DEFAULT_TABLE table=default.Table_d4ce5c2fb18e406f9bcf8759d7bb67b7 [id=c1cdb7a4d210437cbf7f7e03d48978df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.686394 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.686730 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ced4b8365e4fd79c2c5934eb570e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.686735 1163 ts_tablet_manager.cc:1428] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250625 07:52:39.687037 851 leader_election.cc:343] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.687175 1163 tablet_bootstrap.cc:492] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.688679 1163 tablet_bootstrap.cc:654] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.689422 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 52af63b3b42a44ed831f202ff2601073 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.689905 1018 tablet_service.cc:1468] Processing CreateTablet for tablet da3d35622f5845baa3976c5505a00cab (DEFAULT_TABLE table=default.Table_313d5eb00cc14cc882bb2839b715189e [id=e37d990099c14d9da7da93e93a4fe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.691334 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8317ed76301c4e6099172883e30b685a (DEFAULT_TABLE table=default.Table_887824db43f7441e822ae633b51194bd [id=53e10793ef44444188ac72fee6997cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.691841 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8317ed76301c4e6099172883e30b685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.697330 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 68ced4b8365e4fd79c2c5934eb570e7b (DEFAULT_TABLE table=default.Table_d4ce5c2fb18e406f9bcf8759d7bb67b7 [id=c1cdb7a4d210437cbf7f7e03d48978df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.697898 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 83a712d5e38d432188427c277194b3b0 (DEFAULT_TABLE table=default.Table_da1b8e2e55234fae900520bfecbbebe1 [id=6d937f3030514b52ad3d0f19ff90e41c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:39.698294 583 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9b83f4c8544946d9adb3346df200b215 (table default.Table_3aaa21f170574b79ae5d89ff2cc423ad [id=9ae1c8331e2149b3a32cf69a11f834c3], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9b83f4c8544946d9adb3346df200b215 consensus state has no leader
I20250625 07:52:39.699812 2628 raft_consensus.cc:491] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.699864 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.699965 2628 raft_consensus.cc:513] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.700387 850 leader_election.cc:343] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.700547 2628 leader_election.cc:290] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.700600 850 leader_election.cc:304] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.700901 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22411725aa5741c497328ce047c44d50" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.701004 2628 raft_consensus.cc:2747] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:39.701395 851 leader_election.cc:343] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.701826 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22411725aa5741c497328ce047c44d50" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.702227 1163 tablet_bootstrap.cc:492] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.702418 1163 ts_tablet_manager.cc:1397] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20250625 07:52:39.703167 1163 raft_consensus.cc:357] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.703648 1163 raft_consensus.cc:383] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.703754 1163 raft_consensus.cc:738] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.704047 1163 consensus_queue.cc:260] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:39.704819 850 leader_election.cc:343] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.704972 850 leader_election.cc:304] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.705231 2628 raft_consensus.cc:2747] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.705439 1163 ts_tablet_manager.cc:1428] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:39.705873 893 tablet_service.cc:1468] Processing CreateTablet for tablet 83a712d5e38d432188427c277194b3b0 (DEFAULT_TABLE table=default.Table_da1b8e2e55234fae900520bfecbbebe1 [id=6d937f3030514b52ad3d0f19ff90e41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.707089 1163 tablet_bootstrap.cc:492] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.707711 2628 raft_consensus.cc:491] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.708112 2628 raft_consensus.cc:513] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.709012 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a712d5e38d432188427c277194b3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.709323 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f31104720614b028e373ae1500c31cc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.709976 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f31104720614b028e373ae1500c31cc" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.710781 2628 leader_election.cc:290] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.711818 1163 tablet_bootstrap.cc:654] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.712090 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 565d312783f24b9b995595f413ce4697 (DEFAULT_TABLE table=default.Table_b16a97fd65d14358a0b7bc51c14f5f61 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.712486 1013 tablet_service.cc:1468] Processing CreateTablet for tablet d9e73032861a4dc28832fdcc1707bfe0 (DEFAULT_TABLE table=default.Table_fb8fd6ee8d994e9cabf8d6834fc42a1a [id=a4818c3d25a54741bd7807791ccc71ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:39.710789 850 leader_election.cc:343] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.712808 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52af63b3b42a44ed831f202ff2601073. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.712864 851 leader_election.cc:343] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.713099 851 leader_election.cc:304] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.713392 888 tablet_service.cc:1468] Processing CreateTablet for tablet 445096fc2e0242aa8515fc8b321ddadc (DEFAULT_TABLE table=default.Table_ccf4a2bf139a422182f3f71c58ee5e31 [id=c0841d014298472f9535dad35eed27a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.713649 891 tablet_service.cc:1468] Processing CreateTablet for tablet 565d312783f24b9b995595f413ce4697 (DEFAULT_TABLE table=default.Table_b16a97fd65d14358a0b7bc51c14f5f61 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.713761 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445096fc2e0242aa8515fc8b321ddadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.713991 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565d312783f24b9b995595f413ce4697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.712056 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 445096fc2e0242aa8515fc8b321ddadc (DEFAULT_TABLE table=default.Table_ccf4a2bf139a422182f3f71c58ee5e31 [id=c0841d014298472f9535dad35eed27a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.716913 2628 raft_consensus.cc:2747] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.717082 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445096fc2e0242aa8515fc8b321ddadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.718494 749 tablet_service.cc:1468] Processing CreateTablet for tablet 68ced4b8365e4fd79c2c5934eb570e7b (DEFAULT_TABLE table=default.Table_d4ce5c2fb18e406f9bcf8759d7bb67b7 [id=c1cdb7a4d210437cbf7f7e03d48978df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.718845 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ced4b8365e4fd79c2c5934eb570e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.719780 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09cb7d5885654b5b920af2ab442ea59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.720078 888 tablet_service.cc:1468] Processing CreateTablet for tablet d9e73032861a4dc28832fdcc1707bfe0 (DEFAULT_TABLE table=default.Table_fb8fd6ee8d994e9cabf8d6834fc42a1a [id=a4818c3d25a54741bd7807791ccc71ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.720476 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e73032861a4dc28832fdcc1707bfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.721174 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e73032861a4dc28832fdcc1707bfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.725181 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet a41ffd566593499299aba5513ccb2f30 (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:39.726620 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:39.726867 907 raft_consensus.cc:3058] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.729295 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44bfcc2b20ba4d0b83641988f2c0ee35" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.729411 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10149f4ae0645ba88508ca51ae9f77a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.729652 907 raft_consensus.cc:2466] T 0bfa1c4d4b6e4530892118189a5ab516 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:39.729952 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "08cd482ceec244c2bed2b116d5c14d89" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:39.730032 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0bfa1c4d4b6e4530892118189a5ab516" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:39.731074 1050 raft_consensus.cc:3058] T 5199775108e44a9780009e4788a861ad P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:39.738550 1017 tablet_service.cc:1468] Processing CreateTablet for tablet b3a84f7d816148788725216f713b3980 (DEFAULT_TABLE table=default.Table_3e24931f1a5c4745b7a6a7d204738c4e [id=627ae54c9fc94e60a70c03e21597d3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.739029 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a84f7d816148788725216f713b3980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.743093 891 tablet_service.cc:1468] Processing CreateTablet for tablet b3a84f7d816148788725216f713b3980 (DEFAULT_TABLE table=default.Table_3e24931f1a5c4745b7a6a7d204738c4e [id=627ae54c9fc94e60a70c03e21597d3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.743625 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a84f7d816148788725216f713b3980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.746145 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 535259cb581843568b1561c607a7d542 (DEFAULT_TABLE table=default.Table_1389e17927584ad697c9c92802a0aed7 [id=46b6b99547124c50a3cd0a4c6436d13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.746627 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535259cb581843568b1561c607a7d542. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.750569 1163 tablet_bootstrap.cc:492] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.750793 1163 ts_tablet_manager.cc:1397] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.003s
I20250625 07:52:39.751674 1163 raft_consensus.cc:357] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.751936 1163 raft_consensus.cc:383] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.752076 1163 raft_consensus.cc:738] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.752362 1163 consensus_queue.cc:260] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.753053 878 tablet_service.cc:1468] Processing CreateTablet for tablet 535259cb581843568b1561c607a7d542 (DEFAULT_TABLE table=default.Table_1389e17927584ad697c9c92802a0aed7 [id=46b6b99547124c50a3cd0a4c6436d13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.753530 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535259cb581843568b1561c607a7d542. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.755908 2498 raft_consensus.cc:491] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.756109 2498 raft_consensus.cc:513] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.756915 2498 leader_election.cc:290] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.757845 713 leader_election.cc:343] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.758069 713 leader_election.cc:304] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.761015 2498 raft_consensus.cc:2747] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:39.761667 713 leader_election.cc:343] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.762307 713 leader_election.cc:343] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.763862 713 leader_election.cc:343] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.764277 713 leader_election.cc:343] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.764674 713 leader_election.cc:343] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:39.765075 713 leader_election.cc:343] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.765492 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8eea57ec54dd49148c20d79f13e7f926 (DEFAULT_TABLE table=default.Table_78623c0c0cf748428ad86d31b2375690 [id=d12c22a3bc0d4e6b9a21a01baca465eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.765942 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea57ec54dd49148c20d79f13e7f926. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.760337 2407 ts_tablet_manager.cc:1918] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.766207 2407 log.cc:1199] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/716e0c4dab584788bbb7631b4ca934bf
W20250625 07:52:39.766376 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fbb67d7c54b4b96a420883a43bddbe4 consensus state has no leader
I20250625 07:52:39.766551 2407 ts_tablet_manager.cc:1939] T 716e0c4dab584788bbb7631b4ca934bf P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:39.766746 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_db6b195c7fa546a2a9c4532db1a2c058" } modify_external_catalogs: true
I20250625 07:52:39.766903 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_db6b195c7fa546a2a9c4532db1a2c058" } modify_external_catalogs: true
W20250625 07:52:39.768193 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 consensus state has no leader
I20250625 07:52:39.768400 596 master_service.cc:948] ReplaceTablet: received request to replace tablet 71c66d9107024afa8b68deeac22332f6 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:39.768383 752 tablet_service.cc:1468] Processing CreateTablet for tablet 83a712d5e38d432188427c277194b3b0 (DEFAULT_TABLE table=default.Table_da1b8e2e55234fae900520bfecbbebe1 [id=6d937f3030514b52ad3d0f19ff90e41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.768803 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a712d5e38d432188427c277194b3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.769119 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a2c905090ebe4dc2aeeceaf4fcdbff1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.769636 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2c905090ebe4dc2aeeceaf4fcdbff1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.772303 749 tablet_service.cc:1468] Processing CreateTablet for tablet 445096fc2e0242aa8515fc8b321ddadc (DEFAULT_TABLE table=default.Table_ccf4a2bf139a422182f3f71c58ee5e31 [id=c0841d014298472f9535dad35eed27a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.772825 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445096fc2e0242aa8515fc8b321ddadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.772850 2506 raft_consensus.cc:491] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.773008 2506 raft_consensus.cc:513] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.773360 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb17a614c41485f94876570246adcb9 (DEFAULT_TABLE table=default.Table_e2cd58bccd694d1c8d767d21c95e174e [id=f4847ae8c0de42dea395bc33fb759ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.773649 2506 leader_election.cc:290] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.773747 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb17a614c41485f94876570246adcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.773548 588 catalog_manager.cc:5582] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:39.773905 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d7ed91509ba94562a9207c7d7c0d3e2b (table default.Table_ca90d59a48d947778cd899359e52ed41 [id=1353496ce19c4b14bdcc41400f3e9844]). Expected version 1 got 0
I20250625 07:52:39.772503 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2eafe961ca574eec8fe9a444fda4a78e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.774545 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:39.774716 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:39.774885 588 catalog_manager.cc:5582] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:39.775048 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 0669f21a918d49a2a298ecb864f02aec (DEFAULT_TABLE table=default.Table_1b5ed40b96fe4203ab8f5b4ca1ae9a8f [id=b0e1a0fdf6b1494484967b6d49d5888b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.775276 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:39.775599 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669f21a918d49a2a298ecb864f02aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.775491 588 catalog_manager.cc:5582] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
W20250625 07:52:39.774719 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eafe961ca574eec8fe9a444fda4a78e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.776175 588 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:39.779165 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:39.779712 2498 raft_consensus.cc:491] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.779883 2498 raft_consensus.cc:513] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.780457 2498 leader_election.cc:290] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.781422 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet af8c114c1c464bcc95dbd287c88976d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:39.781425 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15766b87fb1b4620970d5fab21465cc4" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:39.781888 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 36ddd8cad9c3416e8577a4e1f639d482 (DEFAULT_TABLE table=default.Table_1857237dcf214e1d949c26f903fc6967 [id=2eb056de2c044d999b5da2b59104c688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.782169 2506 raft_consensus.cc:491] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.782330 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ddd8cad9c3416e8577a4e1f639d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.782359 2506 raft_consensus.cc:513] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.782949 2506 leader_election.cc:290] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:39.783648 986 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:39.785612 749 tablet_service.cc:1468] Processing CreateTablet for tablet d9e73032861a4dc28832fdcc1707bfe0 (DEFAULT_TABLE table=default.Table_fb8fd6ee8d994e9cabf8d6834fc42a1a [id=a4818c3d25a54741bd7807791ccc71ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.785843 985 leader_election.cc:304] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.785974 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e73032861a4dc28832fdcc1707bfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.786468 756 tablet_service.cc:1468] Processing CreateTablet for tablet 52af63b3b42a44ed831f202ff2601073 (DEFAULT_TABLE table=default.Table_2ddae2af7aaf4116a475872071ad289e [id=c476334315c4477ca9a96fd7dd6d2c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.786782 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52af63b3b42a44ed831f202ff2601073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.787400 883 tablet_service.cc:1515] Processing DeleteTablet for tablet d0a75906385744e5ba6938f451a2d7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:39.787775 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0669f21a918d49a2a298ecb864f02aec (DEFAULT_TABLE table=default.Table_1b5ed40b96fe4203ab8f5b4ca1ae9a8f [id=b0e1a0fdf6b1494484967b6d49d5888b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.788236 893 tablet_service.cc:1468] Processing CreateTablet for tablet 36ddd8cad9c3416e8577a4e1f639d482 (DEFAULT_TABLE table=default.Table_1857237dcf214e1d949c26f903fc6967 [id=2eb056de2c044d999b5da2b59104c688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.788373 879 tablet_service.cc:1468] Processing CreateTablet for tablet 8eea57ec54dd49148c20d79f13e7f926 (DEFAULT_TABLE table=default.Table_78623c0c0cf748428ad86d31b2375690 [id=d12c22a3bc0d4e6b9a21a01baca465eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.788653 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669f21a918d49a2a298ecb864f02aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.788885 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ddd8cad9c3416e8577a4e1f639d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.789122 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea57ec54dd49148c20d79f13e7f926. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.789450 1160 ts_tablet_manager.cc:1905] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.789875 2506 raft_consensus.cc:2747] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.791466 1162 tablet_bootstrap.cc:492] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:39.791671 1162 ts_tablet_manager.cc:1397] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.161s user 0.003s sys 0.000s
W20250625 07:52:39.792891 985 leader_election.cc:343] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:39.793154 985 leader_election.cc:304] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.793490 2506 raft_consensus.cc:2747] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.796232 755 tablet_service.cc:1468] Processing CreateTablet for tablet b3a84f7d816148788725216f713b3980 (DEFAULT_TABLE table=default.Table_3e24931f1a5c4745b7a6a7d204738c4e [id=627ae54c9fc94e60a70c03e21597d3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.796510 1162 raft_consensus.cc:357] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.799750 2506 raft_consensus.cc:491] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.799901 2506 raft_consensus.cc:513] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.800596 2506 leader_election.cc:290] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.801307 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb17a614c41485f94876570246adcb9 (DEFAULT_TABLE table=default.Table_e2cd58bccd694d1c8d767d21c95e174e [id=f4847ae8c0de42dea395bc33fb759ca6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:39.801542 986 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 07a07af928ff4d3d894b074ef0cf44eb
W20250625 07:52:39.804319 985 leader_election.cc:343] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:39.804597 985 leader_election.cc:304] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:39.804816 1161 ts_tablet_manager.cc:1918] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:39.804955 2506 raft_consensus.cc:2747] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.805043 1161 log.cc:1199] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/110fe299c7c44df69f459da72a1138a5
I20250625 07:52:39.805446 1161 ts_tablet_manager.cc:1939] T 110fe299c7c44df69f459da72a1138a5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:39.806497 1161 tablet_replica.cc:331] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:39.806761 1161 raft_consensus.cc:2241] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.807096 1161 raft_consensus.cc:2270] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.808612 1163 ts_tablet_manager.cc:1428] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.058s user 0.001s sys 0.003s
I20250625 07:52:39.809101 1163 tablet_bootstrap.cc:492] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.811455 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5598fb3fe5da4acc8468c199fc7618d6 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.811690 883 tablet_service.cc:1468] Processing CreateTablet for tablet 779dd37006244ec28d642441945f98f9 (DEFAULT_TABLE table=default.Table_c688c532bd1047a2b31446259d22c52d [id=476e836459cd41799a5a890d817a4cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.812026 1163 tablet_bootstrap.cc:654] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.813320 885 tablet_service.cc:1468] Processing CreateTablet for tablet 73db63a514504afb977e558ae1698b4e (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.819545 1162 raft_consensus.cc:383] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.819839 1162 raft_consensus.cc:738] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.820356 1162 consensus_queue.cc:260] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.794265 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1351) took 2514 ms. Trace:
I20250625 07:52:39.796718 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3a84f7d816148788725216f713b3980. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.797801 750 tablet_service.cc:1468] Processing CreateTablet for tablet 535259cb581843568b1561c607a7d542 (DEFAULT_TABLE table=default.Table_1389e17927584ad697c9c92802a0aed7 [id=46b6b99547124c50a3cd0a4c6436d13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.798421 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0669f21a918d49a2a298ecb864f02aec (DEFAULT_TABLE table=default.Table_1b5ed40b96fe4203ab8f5b4ca1ae9a8f [id=b0e1a0fdf6b1494484967b6d49d5888b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.798873 754 tablet_service.cc:1468] Processing CreateTablet for tablet 8eea57ec54dd49148c20d79f13e7f926 (DEFAULT_TABLE table=default.Table_78623c0c0cf748428ad86d31b2375690 [id=d12c22a3bc0d4e6b9a21a01baca465eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.799093 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb17a614c41485f94876570246adcb9 (DEFAULT_TABLE table=default.Table_e2cd58bccd694d1c8d767d21c95e174e [id=f4847ae8c0de42dea395bc33fb759ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.823568 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb17a614c41485f94876570246adcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.800940 746 tablet_service.cc:1515] Processing DeleteTablet for tablet f51ef840a3ed40bd939b3fe62df3cda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:39.804477 751 tablet_service.cc:1468] Processing CreateTablet for tablet 36ddd8cad9c3416e8577a4e1f639d482 (DEFAULT_TABLE table=default.Table_1857237dcf214e1d949c26f903fc6967 [id=2eb056de2c044d999b5da2b59104c688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.824841 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ddd8cad9c3416e8577a4e1f639d482. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.821820 1162 ts_tablet_manager.cc:1428] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250625 07:52:39.826431 1162 tablet_bootstrap.cc:492] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:39.805887 743 tablet_service.cc:1468] Processing CreateTablet for tablet 73db63a514504afb977e558ae1698b4e (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.828939 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db63a514504afb977e558ae1698b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.829332 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea57ec54dd49148c20d79f13e7f926. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.832091 1162 tablet_bootstrap.cc:654] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.834164 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0669f21a918d49a2a298ecb864f02aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.835919 1163 tablet_bootstrap.cc:492] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.836172 1163 ts_tablet_manager.cc:1397] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.001s
I20250625 07:52:39.793862 758 tablet_service.cc:1468] Processing CreateTablet for tablet da3d35622f5845baa3976c5505a00cab (DEFAULT_TABLE table=default.Table_313d5eb00cc14cc882bb2839b715189e [id=e37d990099c14d9da7da93e93a4fe260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.806550 747 tablet_service.cc:1468] Processing CreateTablet for tablet 779dd37006244ec28d642441945f98f9 (DEFAULT_TABLE table=default.Table_c688c532bd1047a2b31446259d22c52d [id=476e836459cd41799a5a890d817a4cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.837039 1163 raft_consensus.cc:357] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.837147 2604 raft_consensus.cc:491] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.837261 1163 raft_consensus.cc:383] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.837337 1163 raft_consensus.cc:738] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.837278 2604 raft_consensus.cc:513] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.837427 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779dd37006244ec28d642441945f98f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.837558 1163 consensus_queue.cc:260] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.837803 2604 leader_election.cc:290] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.837903 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535259cb581843568b1561c607a7d542. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.838027 1163 ts_tablet_manager.cc:1428] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:39.838358 1163 tablet_bootstrap.cc:492] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.840024 1163 tablet_bootstrap.cc:654] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.807188 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5598fb3fe5da4acc8468c199fc7618d6 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.840986 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5598fb3fe5da4acc8468c199fc7618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.845297 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3d35622f5845baa3976c5505a00cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.808495 739 tablet_service.cc:1468] Processing CreateTablet for tablet 565d312783f24b9b995595f413ce4697 (DEFAULT_TABLE table=default.Table_b16a97fd65d14358a0b7bc51c14f5f61 [id=b49edf60e62443608091ee1dbc2ae800]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.822018 2407 rpcz_store.cc:276] 0625 07:52:37.279613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.279654 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:39.489789 (+2210135us) ts_tablet_manager.cc:1145] Deleting tablet 716e0c4dab584788bbb7631b4ca934bf
0625 07:52:39.489802 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.699139 (+209337us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.760231 (+ 61092us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.794241 (+ 34010us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26875,"tablet-delete.queue_time_us":2209963}
I20250625 07:52:39.854730 1163 tablet_bootstrap.cc:492] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.855000 1163 ts_tablet_manager.cc:1397] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250625 07:52:39.855854 1163 raft_consensus.cc:357] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.856106 1163 raft_consensus.cc:383] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.856240 1163 raft_consensus.cc:738] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.856518 1163 consensus_queue.cc:260] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.856905 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 110fe299c7c44df69f459da72a1138a5 (table default.Table_55794188bd9d47d29bc77e1c1498512f [id=d281de0430994f13a49c3d53c062dec4]) successfully deleted
I20250625 07:52:39.857141 1163 ts_tablet_manager.cc:1428] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:39.857585 1163 tablet_bootstrap.cc:492] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:39.859549 1163 tablet_bootstrap.cc:654] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:39.863621 2652 raft_consensus.cc:491] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.863966 2652 raft_consensus.cc:513] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.865509 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17be34eba92246df973f3d464e13fd30" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:39.866349 851 leader_election.cc:343] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.866650 2407 tablet_replica.cc:331] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:39.866891 2407 raft_consensus.cc:2241] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:39.867182 2407 raft_consensus.cc:2270] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:39.867797 2407 ts_tablet_manager.cc:1905] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.878844 2652 leader_election.cc:290] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.881160 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17be34eba92246df973f3d464e13fd30" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:39.886656 850 leader_election.cc:343] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.887077 850 leader_election.cc:304] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.888579 2652 raft_consensus.cc:2747] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.890115 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5598fb3fe5da4acc8468c199fc7618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.890648 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb17a614c41485f94876570246adcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.890967 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779dd37006244ec28d642441945f98f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.901763 713 leader_election.cc:343] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 44bfcc2b20ba4d0b83641988f2c0ee35
I20250625 07:52:39.904829 713 leader_election.cc:304] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:39.908372 1163 tablet_bootstrap.cc:492] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:39.909547 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db63a514504afb977e558ae1698b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.915004 2498 raft_consensus.cc:2747] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.927129 1163 ts_tablet_manager.cc:1397] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.006s
I20250625 07:52:39.931008 1163 raft_consensus.cc:357] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.931239 1163 raft_consensus.cc:383] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:39.931304 1163 raft_consensus.cc:738] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:39.931538 1163 consensus_queue.cc:260] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.932013 1163 ts_tablet_manager.cc:1428] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:39.933517 2475 raft_consensus.cc:491] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.933845 2475 raft_consensus.cc:513] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.935267 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:39.935969 851 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:39.937954 2475 leader_election.cc:290] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:39.946844 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565d312783f24b9b995595f413ce4697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.950417 1161 ts_tablet_manager.cc:1905] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:39.953272 2506 raft_consensus.cc:491] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.953487 2506 raft_consensus.cc:513] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.954281 2506 leader_election.cc:290] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.954419 2506 raft_consensus.cc:491] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.954546 2506 raft_consensus.cc:513] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:39.958461 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 73db63a514504afb977e558ae1698b4e (DEFAULT_TABLE table=default.Table_3d1a2916527442ea8dc34a8b834ea1f5 [id=e4c35acfc16b480d9d04f20933197ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.960186 2506 leader_election.cc:290] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.960355 2506 raft_consensus.cc:491] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:39.960474 2506 raft_consensus.cc:513] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:39.961097 2506 leader_election.cc:290] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:39.962795 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
/tmp/dist-test-taskSTcLfj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:43317
--webserver_interface=127.0.104.61
--webserver_port=34053
--builtin_ntp_servers=127.0.104.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250625 07:52:39.965368 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68ced4b8365e4fd79c2c5934eb570e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.965647 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a712d5e38d432188427c277194b3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.975677 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73db63a514504afb977e558ae1698b4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.976123 986 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0657163b9a0d4f7f99f57f7fd1f32462
W20250625 07:52:39.977474 986 leader_election.cc:343] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:39.977741 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 779dd37006244ec28d642441945f98f9 (DEFAULT_TABLE table=default.Table_c688c532bd1047a2b31446259d22c52d [id=476e836459cd41799a5a890d817a4cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:39.978380 1158 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:39.978338 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da3d35622f5845baa3976c5505a00cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:39.980819 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779dd37006244ec28d642441945f98f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.978327 986 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:39.983530 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 565d312783f24b9b995595f413ce4697. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:39.985111 713 leader_election.cc:343] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 08cd482ceec244c2bed2b116d5c14d89
I20250625 07:52:39.988826 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 71c66d9107024afa8b68deeac22332f6
I20250625 07:52:39.989542 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 676c2c66a9e643258fc8745b0f208382
I20250625 07:52:39.989909 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4ea4205b845140cead133d2799fee68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.990021 713 leader_election.cc:304] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:39.990523 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea4205b845140cead133d2799fee68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.990563 2498 raft_consensus.cc:2747] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:39.990983 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c7a05e580684fb2811d89a355b0a26e" } modify_external_catalogs: true
I20250625 07:52:39.991227 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_0c7a05e580684fb2811d89a355b0a26e" } modify_external_catalogs: true
I20250625 07:52:39.991832 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 99acd74c01f34d068c868fde3c6ec1df
I20250625 07:52:39.992024 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_332cfc4acc5744b3b4435e0460b5c13b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:39.992522 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2eb4cc88dc8c42eeb3bdee821d48710d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.993003 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eb4cc88dc8c42eeb3bdee821d48710d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.993050 713 leader_election.cc:304] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:39.993120 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_332cfc4acc5744b3b4435e0460b5c13b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:39.993347 2603 raft_consensus.cc:2802] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:39.993494 2603 raft_consensus.cc:695] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:39.993731 2603 consensus_queue.cc:237] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:39.993925 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27e94c4fe10d48368a212eac8d8d09aa consensus state has no leader
I20250625 07:52:39.994283 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3ddaab806779471489af8e38d15a75d1" } modify_external_catalogs: true
I20250625 07:52:39.994422 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3ddaab806779471489af8e38d15a75d1" } modify_external_catalogs: true
I20250625 07:52:39.995163 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1a052ae2f0774e85bdc42172e61770d1" } modify_external_catalogs: true
I20250625 07:52:39.995355 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1a052ae2f0774e85bdc42172e61770d1" } modify_external_catalogs: true
I20250625 07:52:39.996031 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9ac5f17525cc431e9941ba484be5bdb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:39.996537 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ac5f17525cc431e9941ba484be5bdb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:39.997926 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bdb6ad2e21814a5a829bea1a9bede857 consensus state has no leader
I20250625 07:52:39.998243 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2c51d93cc45a4f19b57e86a55ed454a0" } new_table_name: "default.Table_36647dfbcb404a17abf994a9372c56ed" modify_external_catalogs: true
I20250625 07:52:39.999512 596 catalog_manager.cc:6652] ReplaceTablet: tablet 71c66d9107024afa8b68deeac22332f6 deleted and replaced by tablet ebbda15772854f75ba45d32ca0d8de23
I20250625 07:52:39.999831 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8e925fa4bed3456fb11fa79d527a8a42" } new_table_name: "default.Table_2c0980f39c034333af311d3259ab7270" modify_external_catalogs: true
I20250625 07:52:40.005079 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c83ce536556447e68e849b1839f1a5ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.006130 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c83ce536556447e68e849b1839f1a5ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.019440 713 leader_election.cc:343] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e10149f4ae0645ba88508ca51ae9f77a
I20250625 07:52:40.019608 713 leader_election.cc:304] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.019930 2612 raft_consensus.cc:2747] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.023504 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.023855 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 99acd74c01f34d068c868fde3c6ec1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.024103 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 17be34eba92246df973f3d464e13fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.024394 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 71c66d9107024afa8b68deeac22332f6 with delete_type TABLET_DATA_DELETED (replaced by tablet ebbda15772854f75ba45d32ca0d8de23) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.024725 2475 raft_consensus.cc:491] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.024936 2475 raft_consensus.cc:513] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.028402 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "484a49fab6134d059c64f5a3f2b29e03" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.029117 851 leader_election.cc:343] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.032179 2475 leader_election.cc:290] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:40.032407 1045 raft_consensus.cc:3058] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:40.032799 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5598fb3fe5da4acc8468c199fc7618d6 (DEFAULT_TABLE table=default.Table_7631c038cdc447ce8e361f857334153b [id=7d04c0a20eb540f1b07d86ce5f112bbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.034893 1158 tablet_bootstrap.cc:654] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.038172 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5598fb3fe5da4acc8468c199fc7618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.045187 1160 ts_tablet_manager.cc:1918] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:40.045337 1160 log.cc:1199] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8fdc3220d53142faae41fd1f1500de56
I20250625 07:52:40.045732 1160 ts_tablet_manager.cc:1939] T 8fdc3220d53142faae41fd1f1500de56 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:39.987134 2407 ts_tablet_manager.cc:1918] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:40.045970 2407 log.cc:1199] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:40.046371 2407 ts_tablet_manager.cc:1939] T 5179a144a9a0445f933ca02e79bbc122 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.047219 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1352) took 2767 ms. Trace:
I20250625 07:52:40.047403 2407 rpcz_store.cc:276] 0625 07:52:37.279942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.279978 (+ 36us) service_pool.cc:225] Handling call
0625 07:52:39.853404 (+2573426us) ts_tablet_manager.cc:1145] Deleting tablet 5179a144a9a0445f933ca02e79bbc122
0625 07:52:39.866579 (+ 13175us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:39.872826 (+ 6247us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:39.986805 (+113979us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.047195 (+ 60390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2573244}
I20250625 07:52:40.047833 2407 tablet_replica.cc:331] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.048043 2407 raft_consensus.cc:2241] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.048259 2407 raft_consensus.cc:2270] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.048858 1161 ts_tablet_manager.cc:1918] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.048979 1161 log.cc:1199] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/6e5a207d515449e298936d8bb532437a
I20250625 07:52:40.048914 2407 ts_tablet_manager.cc:1905] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.049497 1161 ts_tablet_manager.cc:1939] T 6e5a207d515449e298936d8bb532437a P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.049803 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1342) took 1170 ms. Trace:
I20250625 07:52:40.049916 1161 rpcz_store.cc:276] 0625 07:52:38.879606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.879649 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:39.806422 (+926773us) ts_tablet_manager.cc:1145] Deleting tablet 6e5a207d515449e298936d8bb532437a
0625 07:52:39.806433 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.018055 (+211622us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.037854 (+ 19799us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.049786 (+ 11932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926633}
I20250625 07:52:40.050355 1161 tablet_replica.cc:331] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:40.051409 1045 raft_consensus.cc:1577] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 2' and following 1 ops. Status for this op: Illegal state: STOPPING
I20250625 07:52:40.052834 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "486463a8140c4e769cc79870d2acff78" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.053998 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 99acd74c01f34d068c868fde3c6ec1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.054123 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 71c66d9107024afa8b68deeac22332f6 with delete_type TABLET_DATA_DELETED (replaced by tablet ebbda15772854f75ba45d32ca0d8de23) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.052862 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "065822c50586488785f7d9d53841a4dd" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.057718 2661 raft_consensus.cc:491] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.057904 2661 raft_consensus.cc:513] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.058329 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 6e5a207d515449e298936d8bb532437a (table default.Table_2908cac0353642cc92bc2152162de70e [id=096ac4617a3446c3bb78ef2f61744fb4]) successfully deleted
I20250625 07:52:40.058576 2661 leader_election.cc:290] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:40.060720 985 leader_election.cc:343] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:40.060897 985 leader_election.cc:304] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.061134 2661 raft_consensus.cc:2747] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.061342 985 leader_election.cc:343] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:40.061460 985 leader_election.cc:304] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.061714 2661 raft_consensus.cc:2747] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.061862 985 leader_election.cc:343] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:40.061941 985 leader_election.cc:304] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.062269 2661 raft_consensus.cc:2747] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.062973 1163 tablet_bootstrap.cc:492] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.064815 1163 tablet_bootstrap.cc:654] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:40.065230 986 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: f4bb74391a1c4e2792226d259a07f977
W20250625 07:52:40.065480 985 leader_election.cc:343] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: f4bb74391a1c4e2792226d259a07f977
I20250625 07:52:40.065591 985 leader_election.cc:304] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.065809 2661 raft_consensus.cc:2747] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.071969 1163 tablet_bootstrap.cc:492] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.072201 1163 ts_tablet_manager.cc:1397] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20250625 07:52:40.072468 1162 tablet_bootstrap.cc:492] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.072613 1161 raft_consensus.cc:2241] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.073084 1163 raft_consensus.cc:357] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.073387 1163 raft_consensus.cc:383] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.073534 1163 raft_consensus.cc:738] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.073858 1161 raft_consensus.cc:2270] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.073868 1163 consensus_queue.cc:260] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.074561 1163 ts_tablet_manager.cc:1428] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:40.075222 1163 tablet_bootstrap.cc:492] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.075457 1162 ts_tablet_manager.cc:1397] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.249s user 0.006s sys 0.010s
I20250625 07:52:40.076781 1162 raft_consensus.cc:357] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.077147 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]) successfully deleted
W20250625 07:52:40.074214 2668 consensus_queue.cc:1162] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250625 07:52:40.079857 1160 tablet_replica.cc:331] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:40.080444 1161 ts_tablet_manager.cc:1905] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.080514 1160 raft_consensus.cc:2241] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.080806 1160 raft_consensus.cc:2270] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.081615 1160 ts_tablet_manager.cc:1905] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.086747 1161 ts_tablet_manager.cc:1918] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:40.081110 1163 tablet_bootstrap.cc:654] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.086896 1161 log.cc:1199] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/abb4d9dc2da7453d8336d5589dda3a74
I20250625 07:52:40.087603 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "484a49fab6134d059c64f5a3f2b29e03" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.087849 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.088227 1161 ts_tablet_manager.cc:1939] T abb4d9dc2da7453d8336d5589dda3a74 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:40.088819 850 leader_election.cc:343] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.089000 850 leader_election.cc:304] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.089917 2475 raft_consensus.cc:2747] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.090425 1162 raft_consensus.cc:383] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.090502 1162 raft_consensus.cc:738] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.091399 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1343) took 1162 ms. Trace:
I20250625 07:52:40.091483 1161 rpcz_store.cc:276] 0625 07:52:38.928923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.929000 (+ 77us) service_pool.cc:225] Handling call
0625 07:52:40.050299 (+1121299us) ts_tablet_manager.cc:1145] Deleting tablet abb4d9dc2da7453d8336d5589dda3a74
0625 07:52:40.050310 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.073737 (+ 23427us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x6160000f5554. stack: 00007f96f846d336 00007f96f846d0f0 00007f96f74633ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9703ecd000 00007f97051df093 00007f9707cbc137 00007f9707cee1c4 00007f9707ccfc20 00007f9707ccf9ad 0000000000a7573c 00007f96f84b29df 00007f96f84b5cab 00007f96f84b5b28
0625 07:52:40.083466 (+ 9729us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.086626 (+ 3160us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.091381 (+ 4755us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50295680,"tablet-delete.queue_time_us":1119443}
I20250625 07:52:40.091835 1161 tablet_replica.cc:331] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:40.091779 850 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.091933 850 leader_election.cc:304] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.092025 1161 raft_consensus.cc:2241] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.092164 1161 raft_consensus.cc:2270] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.092164 1160 ts_tablet_manager.cc:1918] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.092293 1160 log.cc:1199] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/90f2bef7baa54f1a96b3b59921d59457
I20250625 07:52:40.092438 2475 raft_consensus.cc:2747] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.092585 1160 ts_tablet_manager.cc:1939] T 90f2bef7baa54f1a96b3b59921d59457 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:40.092727 1161 ts_tablet_manager.cc:1905] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.093397 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b [id=c80e78f1ef124af2b8f405e7d162ade8]) successfully deleted
W20250625 07:52:40.072929 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 52937us
W20250625 07:52:40.094484 713 leader_election.cc:343] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 065822c50586488785f7d9d53841a4dd
I20250625 07:52:40.096575 2407 ts_tablet_manager.cc:1918] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.096714 2407 log.cc:1199] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:40.098327 1163 tablet_bootstrap.cc:492] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.098562 1163 ts_tablet_manager.cc:1397] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20250625 07:52:40.099714 1163 raft_consensus.cc:357] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.099989 1163 raft_consensus.cc:383] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.100118 1163 raft_consensus.cc:738] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.100448 1163 consensus_queue.cc:260] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.100662 2661 raft_consensus.cc:491] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.100865 2661 raft_consensus.cc:513] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.101569 2661 leader_election.cc:290] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.102094 1163 ts_tablet_manager.cc:1428] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250625 07:52:40.102392 2407 ts_tablet_manager.cc:1939] T af8c114c1c464bcc95dbd287c88976d6 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.102061 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
W20250625 07:52:40.099156 713 leader_election.cc:343] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 486463a8140c4e769cc79870d2acff78
I20250625 07:52:40.101949 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e109c9dca7444681ac517c7fc3e14dfa" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.103152 1163 tablet_bootstrap.cc:492] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.103469 986 leader_election.cc:304] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.103895 2661 raft_consensus.cc:2747] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.104983 1163 tablet_bootstrap.cc:654] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.105334 2672 tablet.cc:1722] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:40.110556 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1344) took 2833 ms. Trace:
I20250625 07:52:40.110668 2407 rpcz_store.cc:276] 0625 07:52:37.277536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.280896 (+ 3360us) service_pool.cc:225] Handling call
0625 07:52:40.047782 (+2766886us) ts_tablet_manager.cc:1145] Deleting tablet af8c114c1c464bcc95dbd287c88976d6
0625 07:52:40.047793 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.088668 (+ 40875us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.096475 (+ 7807us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.110532 (+ 14057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703567}
I20250625 07:52:40.110954 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 90f2bef7baa54f1a96b3b59921d59457 (table default.Table_87e58d9e449a4f69bed998c1f045fbec [id=6a5c9dd32f7e4216b14fea00e94f98d6]) successfully deleted
I20250625 07:52:40.111050 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1371) took 2594 ms. Trace:
I20250625 07:52:40.111141 2407 rpcz_store.cc:276] 0625 07:52:37.517028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.518637 (+ 1609us) service_pool.cc:225] Handling call
0625 07:52:40.110909 (+2592272us) ts_tablet_manager.cc:1145] Deleting tablet 76ed33a26e764dba80b601edf5454600
0625 07:52:40.110921 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.111033 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2592072}
I20250625 07:52:40.111480 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1373) took 2582 ms. Trace:
I20250625 07:52:40.111565 2407 rpcz_store.cc:276] 0625 07:52:37.529439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.530500 (+ 1061us) service_pool.cc:225] Handling call
0625 07:52:40.111376 (+2580876us) ts_tablet_manager.cc:1145] Deleting tablet 841d3a3c98af45339836d9849ff712fd
0625 07:52:40.111388 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.111465 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2580687}
I20250625 07:52:40.112071 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1372) took 2589 ms. Trace:
I20250625 07:52:40.112196 2407 rpcz_store.cc:276] 0625 07:52:37.522328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.526561 (+ 4233us) service_pool.cc:225] Handling call
0625 07:52:40.111788 (+2585227us) ts_tablet_manager.cc:1145] Deleting tablet 5c8b736b762f4b4c996c442970f58a31
0625 07:52:40.111797 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.112056 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553385}
I20250625 07:52:40.112558 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1399) took 2392 ms. Trace:
I20250625 07:52:40.112766 1160 tablet_replica.cc:331] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:40.112756 2407 rpcz_store.cc:276] 0625 07:52:37.720240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.738572 (+ 18332us) service_pool.cc:225] Handling call
0625 07:52:40.112462 (+2373890us) ts_tablet_manager.cc:1145] Deleting tablet a1a354a11c894b8f992f3d3cdd66cc06
0625 07:52:40.112470 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.112543 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373650}
I20250625 07:52:40.112998 1160 raft_consensus.cc:2241] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.113222 1160 raft_consensus.cc:2270] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.113265 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1400) took 2377 ms. Trace:
I20250625 07:52:40.113495 2407 rpcz_store.cc:276] 0625 07:52:37.736379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.738827 (+ 2448us) service_pool.cc:225] Handling call
0625 07:52:40.113157 (+2374330us) ts_tablet_manager.cc:1145] Deleting tablet 9f30a8b59beb4311aca534ed322aff03
0625 07:52:40.113167 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.113248 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2374145}
I20250625 07:52:40.113819 1160 ts_tablet_manager.cc:1905] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.113866 1158 tablet_bootstrap.cc:492] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.113853 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1401) took 2369 ms. Trace:
I20250625 07:52:40.114140 1158 ts_tablet_manager.cc:1397] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.304s user 0.003s sys 0.009s
I20250625 07:52:40.114943 1158 raft_consensus.cc:357] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.115185 1158 raft_consensus.cc:383] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.115248 1158 raft_consensus.cc:738] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.115144 2407 rpcz_store.cc:276] 0625 07:52:37.744491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.749732 (+ 5241us) service_pool.cc:225] Handling call
0625 07:52:40.113757 (+2364025us) ts_tablet_manager.cc:1145] Deleting tablet 0139149bbf574e13be3b0411a83a0593
0625 07:52:40.113765 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.113839 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2363770}
I20250625 07:52:40.115465 1158 consensus_queue.cc:260] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.115972 1158 ts_tablet_manager.cc:1428] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:40.116122 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "034903d699584feb91f35cbb2f62701a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.116458 1158 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.117578 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1398) took 2445 ms. Trace:
I20250625 07:52:40.117672 2407 rpcz_store.cc:276] 0625 07:52:37.672103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.734684 (+ 62581us) service_pool.cc:225] Handling call
0625 07:52:40.117468 (+2382784us) ts_tablet_manager.cc:1145] Deleting tablet 3cc78b37d83b422fae1ed696d45953f6
0625 07:52:40.117479 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.117562 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352508}
I20250625 07:52:40.117781 1050 raft_consensus.cc:3058] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:40.118021 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1404) took 2320 ms. Trace:
I20250625 07:52:40.091004 1162 consensus_queue.cc:260] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.118109 2407 rpcz_store.cc:276] 0625 07:52:37.797752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:37.801898 (+ 4146us) service_pool.cc:225] Handling call
0625 07:52:40.117910 (+2316012us) ts_tablet_manager.cc:1145] Deleting tablet d8ab63bf0a0f4be29cb027505cedf835
0625 07:52:40.117919 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.118006 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2241309}
I20250625 07:52:40.118422 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1407) took 1652 ms. Trace:
I20250625 07:52:40.118485 1162 ts_tablet_manager.cc:1428] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250625 07:52:40.118785 1162 tablet_bootstrap.cc:492] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:40.120293 1162 tablet_bootstrap.cc:654] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.121731 1163 tablet_bootstrap.cc:492] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.118502 2407 rpcz_store.cc:276] 0625 07:52:38.466136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.466188 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:40.118328 (+1652140us) ts_tablet_manager.cc:1145] Deleting tablet d97f4c56eb574432af661966fcda939b
0625 07:52:40.118337 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.118407 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651971}
I20250625 07:52:40.121925 1163 ts_tablet_manager.cc:1397] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20250625 07:52:40.122079 2407 tablet_replica.cc:331] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.122431 2407 raft_consensus.cc:2241] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:40.122756 2407 raft_consensus.cc:2270] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.122737 1163 raft_consensus.cc:357] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.122956 1163 raft_consensus.cc:383] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.123060 1163 raft_consensus.cc:738] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.123350 2407 ts_tablet_manager.cc:1905] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.123317 1163 consensus_queue.cc:260] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.124595 2661 raft_consensus.cc:491] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.124725 1163 ts_tablet_manager.cc:1428] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:40.124747 2661 raft_consensus.cc:513] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.125339 2661 leader_election.cc:290] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.125679 1163 tablet_bootstrap.cc:492] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.125804 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:40.127341 1163 tablet_bootstrap.cc:654] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.128063 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2082e5e28c5c4d1fac7e40acf77f75e6" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.128660 986 leader_election.cc:304] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.129119 2660 raft_consensus.cc:491] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.129241 2660 raft_consensus.cc:513] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.129866 2660 leader_election.cc:290] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.130008 2660 raft_consensus.cc:2747] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.131000 986 leader_election.cc:343] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: fdad7dc1b27e4d87a747642a6d47230c
W20250625 07:52:40.132119 985 leader_election.cc:343] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: fdad7dc1b27e4d87a747642a6d47230c
I20250625 07:52:40.132336 985 leader_election.cc:304] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.134349 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 716e0c4dab584788bbb7631b4ca934bf (table default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]) successfully deleted
I20250625 07:52:40.134894 2661 raft_consensus.cc:2747] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.136623 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ba49ad0cb12d4f738288aea215dbda87
I20250625 07:52:40.140046 594 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:40.141114 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57f90385582f4bb38f2e7dd0287d924b" } new_table_name: "default.Table_ee1efbc24b3b4d95b0580444c890d0bb" modify_external_catalogs: true
I20250625 07:52:40.142191 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9af73a2221464a56a5c9de529f96638e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.142690 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9af73a2221464a56a5c9de529f96638e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.143832 2661 raft_consensus.cc:491] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.143965 2661 raft_consensus.cc:513] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.144574 2661 leader_election.cc:290] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:40.145385 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5951a9bc2b0841fdadf47d8e9a6e3d6b consensus state has no leader
I20250625 07:52:40.145624 596 master_service.cc:948] ReplaceTablet: received request to replace tablet cd226579c77b4938bb031fac0412cbb1 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:40.145850 986 leader_election.cc:343] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c263b671be384cd097041c60306034de
W20250625 07:52:40.146260 985 leader_election.cc:343] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c263b671be384cd097041c60306034de
I20250625 07:52:40.146418 985 leader_election.cc:304] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.146736 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 7b3eafac1b2b46acb2ac6fca087b7f16
I20250625 07:52:40.146833 2661 raft_consensus.cc:2747] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.147504 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b4d68f1d9d8244a2b49f5e1a5f2855f2
W20250625 07:52:40.148061 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2677e6bb2d474390990587e9e9887dfa consensus state has no leader
I20250625 07:52:40.149080 1158 tablet_bootstrap.cc:654] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.152870 2661 raft_consensus.cc:491] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.153016 2661 raft_consensus.cc:513] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.153622 2661 leader_election.cc:290] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:40.154769 986 leader_election.cc:343] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c0cc3ba09d5e4c3d88e74fffc97eea8b
W20250625 07:52:40.155113 985 leader_election.cc:343] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c0cc3ba09d5e4c3d88e74fffc97eea8b
I20250625 07:52:40.155254 985 leader_election.cc:304] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.155516 2661 raft_consensus.cc:2747] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.156787 1158 tablet_bootstrap.cc:492] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.156956 1158 ts_tablet_manager.cc:1397] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250625 07:52:40.160863 1161 ts_tablet_manager.cc:1918] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.166177 1047 raft_consensus.cc:3058] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:40.166801 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ba49ad0cb12d4f738288aea215dbda87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.166848 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.167371 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a93e9198d1417bb1b8e00560921483 with delete_type TABLET_DATA_DELETED (replaced by tablet 99421eff752142348c8a899e7ac961b1) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.167575 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c9aca996884ffeab88381dd3cb189b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.167837 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 328b21f669c442fb86259500e794d222) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.168004 885 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.169785 2407 ts_tablet_manager.cc:1918] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:40.174297 2407 log.cc:1199] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/abb4d9dc2da7453d8336d5589dda3a74
I20250625 07:52:40.174965 2407 ts_tablet_manager.cc:1939] T abb4d9dc2da7453d8336d5589dda3a74 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.176569 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1408) took 1710 ms. Trace:
I20250625 07:52:40.177000 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ba49ad0cb12d4f738288aea215dbda87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.177186 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.177244 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.178164 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 506794253c84486faa926de885c48c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.178810 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.179384 885 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bc347c278c44c69444caf9ded72e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 9b66fad15471497390fc753b4015bb8f) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.179545 2407 rpcz_store.cc:276] 0625 07:52:38.466492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.466536 (+ 44us) service_pool.cc:225] Handling call
0625 07:52:40.122031 (+1655495us) ts_tablet_manager.cc:1145] Deleting tablet abb4d9dc2da7453d8336d5589dda3a74
0625 07:52:40.122041 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.164171 (+ 42130us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.169669 (+ 5498us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.176545 (+ 6876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655282}
I20250625 07:52:40.180059 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1409) took 1713 ms. Trace:
I20250625 07:52:40.181756 1163 tablet_bootstrap.cc:492] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.181893 1163 ts_tablet_manager.cc:1397] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250625 07:52:40.182654 1163 raft_consensus.cc:357] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.182828 1163 raft_consensus.cc:383] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.182883 1163 raft_consensus.cc:738] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.183115 1163 consensus_queue.cc:260] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.186669 1163 ts_tablet_manager.cc:1428] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:40.182925 2407 rpcz_store.cc:276] 0625 07:52:38.466852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.466900 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:40.179923 (+1713023us) ts_tablet_manager.cc:1145] Deleting tablet 05991b31ebf945f29fca8c060b21e7a7
0625 07:52:40.179935 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.180042 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1712874}
I20250625 07:52:40.189906 2407 tablet_replica.cc:331] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.190196 2407 raft_consensus.cc:2241] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.192317 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7fb5f2a41fbf4e11add0bf364fc3628e" } modify_external_catalogs: true
I20250625 07:52:40.192518 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7fb5f2a41fbf4e11add0bf364fc3628e" } modify_external_catalogs: true
I20250625 07:52:40.192818 1161 log.cc:1199] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/faf049714a7e4d7eb947c29f272bbb7c
I20250625 07:52:40.192905 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f862fc2f9357410dae689f37371ee4d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.193404 1161 ts_tablet_manager.cc:1939] T faf049714a7e4d7eb947c29f272bbb7c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:40.193466 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f862fc2f9357410dae689f37371ee4d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.193707 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1344) took 1234 ms. Trace:
I20250625 07:52:40.193782 1161 rpcz_store.cc:276] 0625 07:52:38.959163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.959225 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:40.091785 (+1132560us) ts_tablet_manager.cc:1145] Deleting tablet faf049714a7e4d7eb947c29f272bbb7c
0625 07:52:40.091796 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.146124 (+ 54328us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.160751 (+ 14627us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.193692 (+ 32941us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1132309}
I20250625 07:52:40.194159 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1345) took 1186 ms. Trace:
I20250625 07:52:40.194218 1161 rpcz_store.cc:276] 0625 07:52:39.007478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.007544 (+ 66us) service_pool.cc:225] Handling call
0625 07:52:40.194041 (+1186497us) ts_tablet_manager.cc:1145] Deleting tablet 3488f79d09054f6dbfc6b54c1dee0d6c
0625 07:52:40.194050 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.194143 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186303}
I20250625 07:52:40.194429 1161 tablet_replica.cc:331] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:40.194593 1161 raft_consensus.cc:2241] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.194733 1161 raft_consensus.cc:2270] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.194661 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8a53e4ac425f4b01a2a885cf7ce1c2ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.195132 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a53e4ac425f4b01a2a885cf7ce1c2ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.195240 1161 ts_tablet_manager.cc:1905] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.198655 1158 raft_consensus.cc:357] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.198834 1158 raft_consensus.cc:383] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.198894 1158 raft_consensus.cc:738] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.199082 1158 consensus_queue.cc:260] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.199575 1158 ts_tablet_manager.cc:1428] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.043s user 0.001s sys 0.000s
I20250625 07:52:40.199838 1158 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.200640 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2eafe961ca574eec8fe9a444fda4a78e" } new_table_name: "default.Table_38841e5dae8e4642b89bd495dffe7174" modify_external_catalogs: true
I20250625 07:52:40.201915 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_850acc602efd46efb7b5771d5efa41f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.202412 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850acc602efd46efb7b5771d5efa41f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.203480 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 5179a144a9a0445f933ca02e79bbc122 (table default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]) successfully deleted
I20250625 07:52:40.204627 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.205077 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet faf049714a7e4d7eb947c29f272bbb7c (table default.Table_4c6dba196ca04a6ba70a9180b32eeec3 [id=a1e6b3ecd3bb49ef94da294bb866f824]) successfully deleted
I20250625 07:52:40.205400 1158 tablet_bootstrap.cc:654] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:40.205287 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3488f79d09054f6dbfc6b54c1dee0d6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3488f79d09054f6dbfc6b54c1dee0d6c already in progress: creating tablet
I20250625 07:52:40.206238 2661 raft_consensus.cc:491] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.206359 2661 raft_consensus.cc:513] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.206947 2661 leader_election.cc:290] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.207224 2407 raft_consensus.cc:2270] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:40.207644 985 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:40.207787 2407 ts_tablet_manager.cc:1905] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:40.204210 713 leader_election.cc:343] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.207993 713 leader_election.cc:304] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.208223 2604 raft_consensus.cc:2747] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.209030 713 leader_election.cc:343] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.209069 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_497900c260fb4dbaa34bbeb002b540c9" } modify_external_catalogs: true
I20250625 07:52:40.209131 713 leader_election.cc:304] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.209214 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_497900c260fb4dbaa34bbeb002b540c9" } modify_external_catalogs: true
I20250625 07:52:40.209343 2603 raft_consensus.cc:2747] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.209827 713 leader_election.cc:343] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.210354 2672 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:40.212913 2475 raft_consensus.cc:491] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.213050 2475 raft_consensus.cc:513] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.213609 2475 leader_election.cc:290] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:40.213609 986 leader_election.cc:343] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:40.213725 986 leader_election.cc:304] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.214185 2661 raft_consensus.cc:2747] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.214166 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b04a575e0984f879f0179c8baa3f3fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.214664 851 leader_election.cc:343] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.216986 1163 tablet_bootstrap.cc:492] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:40.219357 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d38e5d7fa884e6eb16d2fcdcb9b65f9 consensus state has no leader
I20250625 07:52:40.221663 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet cd226579c77b4938bb031fac0412cbb1
I20250625 07:52:40.222144 596 catalog_manager.cc:6652] ReplaceTablet: tablet cd226579c77b4938bb031fac0412cbb1 deleted and replaced by tablet 483e104c78d742869fbe2a51460ff100
I20250625 07:52:40.222208 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1b8a59feb1b443b2899089b11bdb6a2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.222401 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_08987651f24242e4a8e9eb2e4223006c" } new_table_name: "default.Table_12da85a976454685aa41d9817323464c" modify_external_catalogs: true
W20250625 07:52:40.222754 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b8a59feb1b443b2899089b11bdb6a2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.223564 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d4e51e8dd19451682fda376228fc6c9 has no consensus state
I20250625 07:52:40.226207 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 486463a8140c4e769cc79870d2acff78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.227512 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 44bfcc2b20ba4d0b83641988f2c0ee35 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.227924 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 065822c50586488785f7d9d53841a4dd with delete_type TABLET_DATA_DELETED (replaced by tablet f51ef840a3ed40bd939b3fe62df3cda2) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.228358 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet cd226579c77b4938bb031fac0412cbb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 483e104c78d742869fbe2a51460ff100) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.228601 885 tablet_service.cc:1515] Processing DeleteTablet for tablet cd226579c77b4938bb031fac0412cbb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 483e104c78d742869fbe2a51460ff100) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.228874 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 71c66d9107024afa8b68deeac22332f6 with delete_type TABLET_DATA_DELETED (replaced by tablet ebbda15772854f75ba45d32ca0d8de23) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.233201 1163 tablet_bootstrap.cc:654] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.234627 1044 raft_consensus.cc:3058] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:40.236853 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f64c42861e60451e92131a435f95b012
W20250625 07:52:40.236958 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9185bc88074a4d69beb1f796eac0428c consensus state has no leader
I20250625 07:52:40.242076 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet f64c42861e60451e92131a435f95b012 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.242287 2475 raft_consensus.cc:491] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.242453 2475 raft_consensus.cc:513] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.243050 2475 leader_election.cc:290] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.243402 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f64c42861e60451e92131a435f95b012 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.243930 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51c584e6bffb4f0a8020dadf78d718a7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.244609 851 leader_election.cc:343] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.246181 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 716e0c4dab584788bbb7631b4ca934bf with delete_type TABLET_DATA_DELETED (replaced by tablet e181a3f1ae2c49349718d842c166b1e6) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.247130 1162 tablet_bootstrap.cc:492] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.247298 1162 ts_tablet_manager.cc:1397] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.129s user 0.006s sys 0.000s
I20250625 07:52:40.248170 1162 raft_consensus.cc:357] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.248389 1162 raft_consensus.cc:383] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.248466 1162 raft_consensus.cc:738] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.248709 1162 consensus_queue.cc:260] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.250065 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 99acd74c01f34d068c868fde3c6ec1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.250469 749 tablet_service.cc:1515] Processing DeleteTablet for tablet e10149f4ae0645ba88508ca51ae9f77a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.250752 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51c584e6bffb4f0a8020dadf78d718a7" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.250864 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 034903d699584feb91f35cbb2f62701a with delete_type TABLET_DATA_DELETED (replaced by tablet 37d228244cca4052889acdc8f1ddd440) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:40.251405 850 leader_election.cc:343] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.251519 850 leader_election.cc:304] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.252014 2475 raft_consensus.cc:2747] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.252492 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 08cd482ceec244c2bed2b116d5c14d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.253026 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b04a575e0984f879f0179c8baa3f3fa" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.253353 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ba49ad0cb12d4f738288aea215dbda87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.253731 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:40.253854 850 leader_election.cc:343] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.253986 850 leader_election.cc:304] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.254086 1158 tablet_bootstrap.cc:492] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.254129 749 tablet_service.cc:1515] Processing DeleteTablet for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.254406 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cd226579c77b4938bb031fac0412cbb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 483e104c78d742869fbe2a51460ff100) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.254393 1158 ts_tablet_manager.cc:1397] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250625 07:52:40.254525 2475 raft_consensus.cc:2747] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.254722 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f64c42861e60451e92131a435f95b012 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.255438 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet af8c114c1c464bcc95dbd287c88976d6 (table default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]) successfully deleted
W20250625 07:52:40.255717 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 76ed33a26e764dba80b601edf5454600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76ed33a26e764dba80b601edf5454600 already in progress: creating tablet
I20250625 07:52:40.255622 1158 raft_consensus.cc:357] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:40.255890 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 841d3a3c98af45339836d9849ff712fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 841d3a3c98af45339836d9849ff712fd already in progress: creating tablet
I20250625 07:52:40.255848 1158 raft_consensus.cc:383] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.255961 1158 raft_consensus.cc:738] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:40.256044 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 5c8b736b762f4b4c996c442970f58a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c8b736b762f4b4c996c442970f58a31 already in progress: creating tablet
W20250625 07:52:40.256186 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet a1a354a11c894b8f992f3d3cdd66cc06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1a354a11c894b8f992f3d3cdd66cc06 already in progress: creating tablet
W20250625 07:52:40.256323 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 9f30a8b59beb4311aca534ed322aff03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f30a8b59beb4311aca534ed322aff03 already in progress: creating tablet
W20250625 07:52:40.256438 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0139149bbf574e13be3b0411a83a0593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0139149bbf574e13be3b0411a83a0593 already in progress: creating tablet
I20250625 07:52:40.256273 1158 consensus_queue.cc:260] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:40.256569 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3cc78b37d83b422fae1ed696d45953f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cc78b37d83b422fae1ed696d45953f6 already in progress: creating tablet
W20250625 07:52:40.256734 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d8ab63bf0a0f4be29cb027505cedf835 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8ab63bf0a0f4be29cb027505cedf835 already in progress: creating tablet
I20250625 07:52:40.259318 1163 tablet_bootstrap.cc:492] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.259450 1163 ts_tablet_manager.cc:1397] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
W20250625 07:52:40.260317 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet d97f4c56eb574432af661966fcda939b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97f4c56eb574432af661966fcda939b already in progress: creating tablet
I20250625 07:52:40.260192 1163 raft_consensus.cc:357] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.260747 1163 raft_consensus.cc:383] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.260829 1163 raft_consensus.cc:738] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.261052 1163 consensus_queue.cc:260] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.261514 1162 ts_tablet_manager.cc:1428] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250625 07:52:40.261584 1163 ts_tablet_manager.cc:1428] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:40.262005 1162 tablet_bootstrap.cc:492] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:40.262449 1163 tablet_bootstrap.cc:492] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.263808 1163 tablet_bootstrap.cc:654] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.265842 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet abb4d9dc2da7453d8336d5589dda3a74 (table default.Table_f52d7ef3e8a04b4fa6b0964d05e5508b [id=c80e78f1ef124af2b8f405e7d162ade8]) successfully deleted
W20250625 07:52:40.266774 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 05991b31ebf945f29fca8c060b21e7a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05991b31ebf945f29fca8c060b21e7a7 already in progress: creating tablet
I20250625 07:52:40.264870 2682 tablet.cc:1722] T 896419bd407f43e1a0a361330ff9821b P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250625 07:52:40.274219 713 leader_election.cc:343] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 034903d699584feb91f35cbb2f62701a
I20250625 07:52:40.274348 713 leader_election.cc:304] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.274964 2612 raft_consensus.cc:2747] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.276201 2612 consensus_queue.cc:1035] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:40.279946 2672 tablet.cc:1722] T 173a565916c1440493ece348599ddba3 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:40.271348 1158 ts_tablet_manager.cc:1428] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250625 07:52:40.284479 1158 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.285216 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 2082e5e28c5c4d1fac7e40acf77f75e6 (table default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00]). Expected version 5 got 0
I20250625 07:52:40.285702 2604 consensus_queue.cc:1035] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:40.272899 1048 raft_consensus.cc:3058] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:40.286649 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cc1842f257d409ba0a56873f582f023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:40.287002 713 consensus_peers.cc:487] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fdc3220d53142faae41fd1f1500de56. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:40.287169 1158 tablet_bootstrap.cc:654] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.279996 2688 tablet.cc:1722] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:40.292997 713 consensus_peers.cc:487] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:40.293188 713 consensus_peers.cc:487] T ffd98b2877b944f0bd484efa2a38a00a P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:40.298456 2475 raft_consensus.cc:491] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.298641 2475 raft_consensus.cc:513] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.299320 2475 leader_election.cc:290] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:40.299870 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.300387 851 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.300740 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:40.301167 850 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.301266 850 leader_election.cc:304] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.301779 2691 tablet.cc:1722] T e109c9dca7444681ac517c7fc3e14dfa P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:40.302232 2475 raft_consensus.cc:2747] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.305379 1162 tablet_bootstrap.cc:654] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.306988 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet e109c9dca7444681ac517c7fc3e14dfa (table default.Table_144eb7a6f3a1438294332f0eded4a470 [id=eb23c86687b2498e8b2bbb0f3bb37db4]). Expected version 3 got 0
I20250625 07:52:40.307202 588 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:40.311578 2407 ts_tablet_manager.cc:1918] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.311762 2407 log.cc:1199] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/84f47caf3bae41b79b0aa1e320f15da4
I20250625 07:52:40.312172 2407 ts_tablet_manager.cc:1939] T 84f47caf3bae41b79b0aa1e320f15da4 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.315678 2692 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:40.329994 713 consensus_peers.cc:487] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:40.331382 1162 tablet_bootstrap.cc:492] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.332252 1162 ts_tablet_manager.cc:1397] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.004s
I20250625 07:52:40.335981 1162 raft_consensus.cc:357] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.336782 1162 raft_consensus.cc:383] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.331450 2689 tablet.cc:1722] T cb23f900faf6421fb6e72f2b5202d0fd P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:40.337642 1161 ts_tablet_manager.cc:1918] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.337764 1161 log.cc:1199] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/086c9321a70e44b5bb3ea419bb466fbe
I20250625 07:52:40.337914 1162 raft_consensus.cc:738] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.338117 1161 ts_tablet_manager.cc:1939] T 086c9321a70e44b5bb3ea419bb466fbe P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.338471 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1346) took 1288 ms. Trace:
I20250625 07:52:40.338408 1162 consensus_queue.cc:260] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.338567 1161 rpcz_store.cc:276] 0625 07:52:39.049670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.049737 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:40.194394 (+1144657us) ts_tablet_manager.cc:1145] Deleting tablet 086c9321a70e44b5bb3ea419bb466fbe
0625 07:52:40.194401 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.318625 (+124224us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.337523 (+ 18898us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.338451 (+ 928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1144473}
I20250625 07:52:40.338884 1161 tablet_replica.cc:331] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:40.339071 1161 raft_consensus.cc:2241] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.339237 1161 raft_consensus.cc:2270] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.339782 1161 ts_tablet_manager.cc:1905] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.340287 2696 tablet.cc:1722] T 173a565916c1440493ece348599ddba3 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:40.344381 1163 tablet_bootstrap.cc:492] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.344530 1163 ts_tablet_manager.cc:1397] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.003s
I20250625 07:52:40.345022 2688 tablet.cc:1717] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Skipping requested alter to schema version 2, tablet already version 2
I20250625 07:52:40.345256 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 086c9321a70e44b5bb3ea419bb466fbe (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]) successfully deleted
I20250625 07:52:40.345412 1163 raft_consensus.cc:357] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.345611 1163 raft_consensus.cc:383] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.345685 1163 raft_consensus.cc:738] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.345921 1163 consensus_queue.cc:260] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.346731 1163 ts_tablet_manager.cc:1428] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:40.347052 1163 tablet_bootstrap.cc:492] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.348093 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7e5a70063453457485e565593de59bec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.348394 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9afbd980bdec4740a7f68571c5a4fe5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.348644 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e5a70063453457485e565593de59bec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.348846 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9afbd980bdec4740a7f68571c5a4fe5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.350116 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57546ca5ffb04419aaa4086e381744e1" } modify_external_catalogs: true
I20250625 07:52:40.350262 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_57546ca5ffb04419aaa4086e381744e1" } modify_external_catalogs: true
I20250625 07:52:40.354962 1160 ts_tablet_manager.cc:1918] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.355087 1160 log.cc:1199] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/26db0effa17b4cacb1450a01f9631ddf
I20250625 07:52:40.355386 1160 ts_tablet_manager.cc:1939] T 26db0effa17b4cacb1450a01f9631ddf P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:40.356369 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 26db0effa17b4cacb1450a01f9631ddf (table default.Table_f00b99b9279946bab7c178245d31c414 [id=5b100b36de77408ebb66b9d7cd64afb6]) successfully deleted
I20250625 07:52:40.357520 1158 tablet_bootstrap.cc:492] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.358310 1158 ts_tablet_manager.cc:1397] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20250625 07:52:40.359489 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9e0c2aded59d426c9ce157d9d9038958" } modify_external_catalogs: true
I20250625 07:52:40.359670 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9e0c2aded59d426c9ce157d9d9038958" } modify_external_catalogs: true
I20250625 07:52:40.359879 1158 raft_consensus.cc:357] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.360037 1158 raft_consensus.cc:383] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.360090 1158 raft_consensus.cc:738] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.360184 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fd84ef6e2ba44f1ebc48f42fcdbc7d37" } new_table_name: "default.Table_041d5b9fa5c14020b2e1034c466170c0" modify_external_catalogs: true
I20250625 07:52:40.360342 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b1e0cba0d5f340a88739f9994705b2e7" } modify_external_catalogs: true
I20250625 07:52:40.360299 1158 consensus_queue.cc:260] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.360472 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b1e0cba0d5f340a88739f9994705b2e7" } modify_external_catalogs: true
I20250625 07:52:40.360824 1158 ts_tablet_manager.cc:1428] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:40.361079 1158 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.361443 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5976d462ca1440b6b16932b369bfa86a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.361912 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5976d462ca1440b6b16932b369bfa86a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.362725 1158 tablet_bootstrap.cc:654] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.364861 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4e667e326e6e46eb82ccbda12435479a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.365355 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e667e326e6e46eb82ccbda12435479a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.370164 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_327aebe35f38451ba0f50824bef20577"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.370711 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_327aebe35f38451ba0f50824bef20577 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.372767 1163 tablet_bootstrap.cc:654] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:40.383994 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e529cb070cbf4272b900484ce268b291 consensus state has no leader
I20250625 07:52:40.386308 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d320734990404d1e9222bea1d88a18fe
I20250625 07:52:40.387732 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d320734990404d1e9222bea1d88a18fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.388238 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet d320734990404d1e9222bea1d88a18fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.390830 2475 raft_consensus.cc:491] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.390974 2475 raft_consensus.cc:513] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.391566 2475 leader_election.cc:290] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.392174 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5bc347c278c44c69444caf9ded72e1a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.392630 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_143a766c3eb149338d9b90575b6deaf1" } new_table_name: "default.Table_629ca82911b747e8af6c63294a5aae21" modify_external_catalogs: true
W20250625 07:52:40.392741 850 leader_election.cc:343] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.397817 1158 tablet_bootstrap.cc:492] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.398006 1158 ts_tablet_manager.cc:1397] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250625 07:52:40.398928 1158 raft_consensus.cc:357] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.399129 1158 raft_consensus.cc:383] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.399187 1158 raft_consensus.cc:738] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:40.399777 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46f81617a460458791e215b81c589241 consensus state has no leader
I20250625 07:52:40.404884 1158 consensus_queue.cc:260] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.405567 1158 ts_tablet_manager.cc:1428] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250625 07:52:40.405787 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 35f819ef1fa74982a50d6714de305535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f819ef1fa74982a50d6714de305535 already in progress: creating tablet
I20250625 07:52:40.406910 1158 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.409251 2612 consensus_queue.cc:1035] T d7ed91509ba94562a9207c7d7c0d3e2b P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:40.409528 1162 ts_tablet_manager.cc:1428] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20250625 07:52:40.409737 2604 raft_consensus.cc:491] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.409848 1162 tablet_bootstrap.cc:492] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:40.410815 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5bc347c278c44c69444caf9ded72e1a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.411423 1161 ts_tablet_manager.cc:1918] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.411497 1162 tablet_bootstrap.cc:654] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:40.411621 851 leader_election.cc:343] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.411749 851 leader_election.cc:304] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.412158 2475 raft_consensus.cc:2747] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.412596 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1410) took 1945 ms. Trace:
I20250625 07:52:40.412671 2407 rpcz_store.cc:276] 0625 07:52:38.467211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.467258 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:40.188942 (+1721684us) ts_tablet_manager.cc:1145] Deleting tablet 84f47caf3bae41b79b0aa1e320f15da4
0625 07:52:40.188953 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.294199 (+105246us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.311456 (+ 17257us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.412577 (+101121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61594,"tablet-delete.queue_time_us":1721531}
I20250625 07:52:40.413040 2407 tablet_replica.cc:331] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.413214 2407 raft_consensus.cc:2241] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:40.413359 1163 tablet_bootstrap.cc:492] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.413427 2407 pending_rounds.cc:70] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:40.413486 1163 ts_tablet_manager.cc:1397] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250625 07:52:40.413463 2407 pending_rounds.cc:77] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432276128198656 op_type: NO_OP noop_request { }
I20250625 07:52:40.413550 2407 raft_consensus.cc:2887] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:40.413626 2407 raft_consensus.cc:2270] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.409854 2604 raft_consensus.cc:513] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.414187 2407 ts_tablet_manager.cc:1905] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.414343 2604 leader_election.cc:290] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.414331 1163 raft_consensus.cc:357] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.414489 1163 raft_consensus.cc:383] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.414541 1163 raft_consensus.cc:738] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.414745 1163 consensus_queue.cc:260] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.415728 1163 ts_tablet_manager.cc:1428] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:40.416064 1163 tablet_bootstrap.cc:492] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:40.417776 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4dbf323dfa034cc78904cb73077f4ccf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbf323dfa034cc78904cb73077f4ccf already in progress: creating tablet
W20250625 07:52:40.419128 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f51ef840a3ed40bd939b3fe62df3cda2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ef840a3ed40bd939b3fe62df3cda2 already in progress: creating tablet
I20250625 07:52:40.419500 1160 tablet_replica.cc:331] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:40.419761 1160 raft_consensus.cc:2241] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.419975 1160 raft_consensus.cc:2270] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.420550 1160 ts_tablet_manager.cc:1905] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.420814 1161 log.cc:1199] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/97b9a14151fc472e956126620633be32
I20250625 07:52:40.421269 1161 ts_tablet_manager.cc:1939] T 97b9a14151fc472e956126620633be32 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.412554 1158 tablet_bootstrap.cc:654] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.423112 2407 ts_tablet_manager.cc:1918] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:40.423285 2407 log.cc:1199] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:40.423547 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1347) took 1373 ms. Trace:
I20250625 07:52:40.423740 2407 ts_tablet_manager.cc:1939] T 2bcfe7c3c54741fbb0aca69650fc8eee P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.423669 1161 rpcz_store.cc:276] 0625 07:52:39.049853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.049904 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:40.338838 (+1288934us) ts_tablet_manager.cc:1145] Deleting tablet 97b9a14151fc472e956126620633be32
0625 07:52:40.338847 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.381494 (+ 42647us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.411307 (+ 29813us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.423528 (+ 12221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288779}
I20250625 07:52:40.424140 1163 tablet_bootstrap.cc:654] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.424113 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 97b9a14151fc472e956126620633be32 (table default.Table_efc0d4ed62e24872b76dc943522353cd [id=ed2a958c87f045848b9d89d26e79d89a]) successfully deleted
I20250625 07:52:40.424795 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1411) took 1957 ms. Trace:
I20250625 07:52:40.424875 2407 rpcz_store.cc:276] 0625 07:52:38.467532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.467578 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:40.412992 (+1945414us) ts_tablet_manager.cc:1145] Deleting tablet 2bcfe7c3c54741fbb0aca69650fc8eee
0625 07:52:40.413002 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.418824 (+ 5822us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.423012 (+ 4188us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.424770 (+ 1758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945268}
I20250625 07:52:40.425180 2407 tablet_replica.cc:331] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.425343 2407 raft_consensus.cc:2241] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.425467 2407 raft_consensus.cc:2270] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.425946 2407 ts_tablet_manager.cc:1905] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.427112 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c1a0b0bfc2444e53bcb089f929de637f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.427639 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1a0b0bfc2444e53bcb089f929de637f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.428895 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f5b05c08da4a4058ad3334ce9b0a8a84" } new_table_name: "default.Table_d537e64b1f534e34ae1901dc0ca3154f" modify_external_catalogs: true
W20250625 07:52:40.412720 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 79490.2us
I20250625 07:52:40.430660 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d8490c9398a741279aa6a5633cd0387a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.430711 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_071bd7e6cf2a4822b9a762cae07ceb4e" } new_table_name: "default.Table_bc60ae6057e24cb8aa07fc94a678c0d9" modify_external_catalogs: true
W20250625 07:52:40.431133 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8490c9398a741279aa6a5633cd0387a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.431594 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1348) took 1333 ms. Trace:
I20250625 07:52:40.431681 1161 rpcz_store.cc:276] 0625 07:52:39.097724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.097866 (+ 142us) service_pool.cc:225] Handling call
0625 07:52:40.431444 (+1333578us) ts_tablet_manager.cc:1145] Deleting tablet 60c01f0a75674c469b41fac638799afa
0625 07:52:40.431458 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.431571 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333377}
I20250625 07:52:40.431818 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9b7eb79f055b4d6ba3bb32e5c61bdf04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.432137 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 60c01f0a75674c469b41fac638799afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c01f0a75674c469b41fac638799afa already in progress: creating tablet
W20250625 07:52:40.432247 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b7eb79f055b4d6ba3bb32e5c61bdf04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.434521 1161 tablet_replica.cc:331] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:40.434767 1161 raft_consensus.cc:2241] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:40.438221 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45855a33616e441b965dc8b14ab4b4bf consensus state has no leader
I20250625 07:52:40.439142 1161 raft_consensus.cc:2270] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.440109 596 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:40.440841 1161 ts_tablet_manager.cc:1905] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.442102 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_212082b707ae4f5689a2dcffa943a19c" } modify_external_catalogs: true
I20250625 07:52:40.442273 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_212082b707ae4f5689a2dcffa943a19c" } modify_external_catalogs: true
I20250625 07:52:40.443523 1162 tablet_bootstrap.cc:492] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.443737 1162 ts_tablet_manager.cc:1397] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.003s
I20250625 07:52:40.444548 1162 raft_consensus.cc:357] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.444814 1162 raft_consensus.cc:383] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.444929 1162 raft_consensus.cc:738] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.445277 1162 consensus_queue.cc:260] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.446424 1162 ts_tablet_manager.cc:1428] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:40.446696 1162 tablet_bootstrap.cc:492] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:40.448500 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37d228244cca4052889acdc8f1ddd440 consensus state has no leader
I20250625 07:52:40.448357 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a47f7fe519514bb0a711481e766950b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.448668 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bf663ee0009a49b6839fc491f911313e" } new_table_name: "default.Table_1cac7df2b2954c6996da3202ba19ea4c" modify_external_catalogs: true
W20250625 07:52:40.449000 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a47f7fe519514bb0a711481e766950b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.451249 1162 tablet_bootstrap.cc:654] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.453369 1158 tablet_bootstrap.cc:492] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.453510 1158 ts_tablet_manager.cc:1397] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20250625 07:52:40.454243 1158 raft_consensus.cc:357] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.454411 1158 raft_consensus.cc:383] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.454459 1158 raft_consensus.cc:738] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.454684 1158 consensus_queue.cc:260] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.455291 1158 ts_tablet_manager.cc:1428] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:40.455581 1158 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.458914 1158 tablet_bootstrap.cc:654] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.461714 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_32b28c84687545e0aab58d60861754c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.462432 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b28c84687545e0aab58d60861754c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.466161 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28dddfa84bc1497a962c718dad7d3b6b consensus state has no leader
I20250625 07:52:40.471218 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d320734990404d1e9222bea1d88a18fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.474174 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 84f47caf3bae41b79b0aa1e320f15da4 (table default.Table_be7ed090191c4152b6e6bf3651b6988b [id=338b6798f70942bcacc38a84d304e31d]) successfully deleted
I20250625 07:52:40.481076 1161 ts_tablet_manager.cc:1918] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.481937 1161 log.cc:1199] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/d393a85f5fde4b3fb57e5b3ec372791c
I20250625 07:52:40.482929 1162 tablet_bootstrap.cc:492] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.483129 1162 ts_tablet_manager.cc:1397] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20250625 07:52:40.483819 1161 ts_tablet_manager.cc:1939] T d393a85f5fde4b3fb57e5b3ec372791c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.485666 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1349) took 1358 ms. Trace:
I20250625 07:52:40.486346 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a396bde4b5964697ad0090eab777ba0e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.488241 1161 rpcz_store.cc:276] 0625 07:52:39.127458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.127514 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:40.434470 (+1306956us) ts_tablet_manager.cc:1145] Deleting tablet d393a85f5fde4b3fb57e5b3ec372791c
0625 07:52:40.434480 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.454825 (+ 20345us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.480948 (+ 26123us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.485648 (+ 4700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1306727}
I20250625 07:52:40.488564 1161 tablet_replica.cc:331] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:40.488665 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet d393a85f5fde4b3fb57e5b3ec372791c (table default.Table_187c1ac9eb3649369d0c23cfd8eb18e8 [id=023c42acf83a4e5bb295efc76739a3e9]) successfully deleted
I20250625 07:52:40.484642 1162 raft_consensus.cc:357] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.488348 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 2bcfe7c3c54741fbb0aca69650fc8eee (table default.Table_aa09e516c85c4bc0abcb6bead66d3be1 [id=0d08c43a2d7a432391933600b06ce3fe]) successfully deleted
I20250625 07:52:40.489272 1162 raft_consensus.cc:383] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.489717 1162 raft_consensus.cc:738] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.489939 1163 tablet_bootstrap.cc:492] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.490082 1163 ts_tablet_manager.cc:1397] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20250625 07:52:40.490386 1162 consensus_queue.cc:260] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.490818 1161 raft_consensus.cc:2241] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.492146 1161 raft_consensus.cc:2270] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.492749 2661 raft_consensus.cc:491] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.492888 2661 raft_consensus.cc:513] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.493217 1161 ts_tablet_manager.cc:1905] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.493499 2661 leader_election.cc:290] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.494091 1163 raft_consensus.cc:357] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.494346 1163 raft_consensus.cc:383] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250625 07:52:40.494328 986 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9cc1842f257d409ba0a56873f582f023
I20250625 07:52:40.494467 1163 raft_consensus.cc:738] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:40.494812 985 leader_election.cc:343] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9cc1842f257d409ba0a56873f582f023
I20250625 07:52:40.494793 1163 consensus_queue.cc:260] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.494915 985 leader_election.cc:304] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.495124 2661 raft_consensus.cc:2747] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.495510 1163 ts_tablet_manager.cc:1428] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:40.496116 1163 tablet_bootstrap.cc:492] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.492653 1162 ts_tablet_manager.cc:1428] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250625 07:52:40.497078 1162 tablet_bootstrap.cc:492] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:40.497784 1163 tablet_bootstrap.cc:654] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.499692 1162 tablet_bootstrap.cc:654] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.507409 2661 raft_consensus.cc:491] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.510159 1160 ts_tablet_manager.cc:1918] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:40.510294 1160 log.cc:1199] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e4b87309b76f4f0e9dde120d9f755c44
I20250625 07:52:40.510622 1160 ts_tablet_manager.cc:1939] T e4b87309b76f4f0e9dde120d9f755c44 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:40.510730 1162 tablet_bootstrap.cc:492] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.510885 1162 ts_tablet_manager.cc:1397] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250625 07:52:40.511471 1160 tablet_replica.cc:331] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:40.511667 1160 raft_consensus.cc:2241] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.511647 1162 raft_consensus.cc:357] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.511806 1160 raft_consensus.cc:2270] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.511824 1162 raft_consensus.cc:383] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.511883 1162 raft_consensus.cc:738] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.512094 2660 raft_consensus.cc:491] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.512116 1162 consensus_queue.cc:260] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.512233 2660 raft_consensus.cc:513] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.512869 1162 ts_tablet_manager.cc:1428] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:40.513000 2660 leader_election.cc:290] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.513180 1162 tablet_bootstrap.cc:492] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
W20250625 07:52:40.513911 986 leader_election.cc:343] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:40.515194 1162 tablet_bootstrap.cc:654] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.516680 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91]) successfully deleted
I20250625 07:52:40.517551 1163 tablet_bootstrap.cc:492] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.517740 1163 ts_tablet_manager.cc:1397] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20250625 07:52:40.518635 1163 raft_consensus.cc:357] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.518865 1163 raft_consensus.cc:383] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.518965 1163 raft_consensus.cc:738] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.519220 1163 consensus_queue.cc:260] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.519974 1163 ts_tablet_manager.cc:1428] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:40.520278 1163 tablet_bootstrap.cc:492] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.521842 1160 ts_tablet_manager.cc:1905] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.523101 1162 tablet_bootstrap.cc:492] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:40.523255 1162 ts_tablet_manager.cc:1397] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20250625 07:52:40.523995 1163 tablet_bootstrap.cc:654] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.523972 1162 raft_consensus.cc:357] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.524143 1162 raft_consensus.cc:383] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.526062 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 65231ee1a59d49eb96ba352b74c3460c (table default.Table_37564180b46a46b7bc77bf339f37cafb [id=dae978f4a15b4e4ba338dd62912614b8]). Expected version 2 got 0
I20250625 07:52:40.526114 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:40.528146 1158 tablet_bootstrap.cc:492] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.528298 1158 ts_tablet_manager.cc:1397] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250625 07:52:40.528246 591 catalog_manager.cc:5582] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:40.528604 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_3e4796a533d44263a8627ae547f7c8ba [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 3 got 0
I20250625 07:52:40.529035 591 catalog_manager.cc:5582] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:40.529354 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:40.529654 591 catalog_manager.cc:5582] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:40.529934 591 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:40.530373 2661 raft_consensus.cc:513] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.530481 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_5c37d7ee9cf14b9b80231a5991ce9acf [id=38f8a949076e4a64acdda251b315ac87] alter complete (version 2)
I20250625 07:52:40.531023 2661 leader_election.cc:290] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:40.536516 985 leader_election.cc:343] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:40.536585 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 1bbf1cfac4e14a20b3fb53607cdf100f
I20250625 07:52:40.536656 985 leader_election.cc:304] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.536921 2661 raft_consensus.cc:2747] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.537894 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6d6b4bca32de4a538071a652409e93b2
I20250625 07:52:40.538267 1013 tablet_service.cc:1468] Processing CreateTablet for tablet ed4166d5cece4934a0dfb0c9cf3cc690 (DEFAULT_TABLE table=default.Table_880f1c0b45ed4ab38c040de1a705f41c [id=422648aa29d3412f9b1653ba53b95324]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.538661 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4166d5cece4934a0dfb0c9cf3cc690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.540302 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 47f0bee21bf441d99fccdf823cb7be07 (DEFAULT_TABLE table=default.Table_4970ea915b7042629008e088f9d45013 [id=21b03aeae46d444b9e561294fa86e3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.535614 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:40.547866 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6b4bca32de4a538071a652409e93b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.539474 885 tablet_service.cc:1468] Processing CreateTablet for tablet ed4166d5cece4934a0dfb0c9cf3cc690 (DEFAULT_TABLE table=default.Table_880f1c0b45ed4ab38c040de1a705f41c [id=422648aa29d3412f9b1653ba53b95324]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.541343 886 tablet_service.cc:1468] Processing CreateTablet for tablet 47f0bee21bf441d99fccdf823cb7be07 (DEFAULT_TABLE table=default.Table_4970ea915b7042629008e088f9d45013 [id=21b03aeae46d444b9e561294fa86e3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.528036 2498 consensus_queue.cc:1035] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250625 07:52:40.542040 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.542738 879 tablet_service.cc:1468] Processing CreateTablet for tablet ca96d73ad9be44f29ea7fdfcf9132951 (DEFAULT_TABLE table=default.Table_08fde6e3798a4aa0ab2006ff6532dd63 [id=8aa99a74daeb4d58a948d8e78e4cc709]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.543987 883 tablet_service.cc:1468] Processing CreateTablet for tablet 839f99e19a3341cb90792caae491b3b4 (DEFAULT_TABLE table=default.Table_a59eca497c7e423b801c1f7e4e1ed131 [id=fde83281868e41bbbf7df57ba3225be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.524211 1162 raft_consensus.cc:738] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.548668 889 tablet_service.cc:1468] Processing CreateTablet for tablet f925f53350b14044bff3f3f797641e4e (DEFAULT_TABLE table=default.Table_5bcfcafad68c45a28e30214ffb855371 [id=7966682bd34146818413026936a049ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.548947 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4166d5cece4934a0dfb0c9cf3cc690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.548961 1162 consensus_queue.cc:260] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.549177 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f99e19a3341cb90792caae491b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.549322 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca96d73ad9be44f29ea7fdfcf9132951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.549425 893 tablet_service.cc:1468] Processing CreateTablet for tablet 05b952ac910c4734bda9edd17efafda0 (DEFAULT_TABLE table=default.Table_87be1efdccf045a9877e5f62907de1f6 [id=98ba61c1ea47423982578ed6ae3f7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.549602 1162 ts_tablet_manager.cc:1428] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250625 07:52:40.549649 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f0bee21bf441d99fccdf823cb7be07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.550400 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0377a1e4099345e292541aeacd2d07ff (DEFAULT_TABLE table=default.Table_114f4c0ed74b446b9ddf211e56589e6b [id=e8351eff387846b4b1a094ba851f724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.551100 888 tablet_service.cc:1468] Processing CreateTablet for tablet 82aa89659db14725a910cc917ed9a7f9 (DEFAULT_TABLE table=default.Table_7d8f8cc6cbff4a34b036a097f4fa40ce [id=bdc819b1871c4e82b290288da6b2eaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.552605 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b952ac910c4734bda9edd17efafda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.552925 891 tablet_service.cc:1468] Processing CreateTablet for tablet 6f14b21749ab4b0b95a7662c190b2e62 (DEFAULT_TABLE table=default.Table_330c8b7943384fccb11c9490fd7dd310 [id=ff627b4ecef449f39f73038d27da68b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.553222 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f14b21749ab4b0b95a7662c190b2e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.556916 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0377a1e4099345e292541aeacd2d07ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.557169 1161 ts_tablet_manager.cc:1918] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.557272 1161 log.cc:1199] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/f4bb74391a1c4e2792226d259a07f977
I20250625 07:52:40.557529 1161 ts_tablet_manager.cc:1939] T f4bb74391a1c4e2792226d259a07f977 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.561025 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f925f53350b14044bff3f3f797641e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.561167 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.561683 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ca96d73ad9be44f29ea7fdfcf9132951 (DEFAULT_TABLE table=default.Table_08fde6e3798a4aa0ab2006ff6532dd63 [id=8aa99a74daeb4d58a948d8e78e4cc709]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.562075 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 0377a1e4099345e292541aeacd2d07ff (DEFAULT_TABLE table=default.Table_114f4c0ed74b446b9ddf211e56589e6b [id=e8351eff387846b4b1a094ba851f724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.562048 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 839f99e19a3341cb90792caae491b3b4 (DEFAULT_TABLE table=default.Table_a59eca497c7e423b801c1f7e4e1ed131 [id=fde83281868e41bbbf7df57ba3225be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.562681 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 82aa89659db14725a910cc917ed9a7f9 (DEFAULT_TABLE table=default.Table_7d8f8cc6cbff4a34b036a097f4fa40ce [id=bdc819b1871c4e82b290288da6b2eaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.563288 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6f14b21749ab4b0b95a7662c190b2e62 (DEFAULT_TABLE table=default.Table_330c8b7943384fccb11c9490fd7dd310 [id=ff627b4ecef449f39f73038d27da68b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.563724 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6b4bca32de4a538071a652409e93b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.564045 1014 tablet_service.cc:1468] Processing CreateTablet for tablet f925f53350b14044bff3f3f797641e4e (DEFAULT_TABLE table=default.Table_5bcfcafad68c45a28e30214ffb855371 [id=7966682bd34146818413026936a049ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.564390 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 05b952ac910c4734bda9edd17efafda0 (DEFAULT_TABLE table=default.Table_87be1efdccf045a9877e5f62907de1f6 [id=98ba61c1ea47423982578ed6ae3f7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.567227 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aa89659db14725a910cc917ed9a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.575479 1022 tablet_service.cc:1468] Processing CreateTablet for tablet edc15abe1a4b46b693167c8fda7f9ce1 (DEFAULT_TABLE table=default.Table_1bb389d88a7c49ea86fc830f2393cb07 [id=b8ac486798e449c49221196462acbe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.576334 879 tablet_service.cc:1468] Processing CreateTablet for tablet edc15abe1a4b46b693167c8fda7f9ce1 (DEFAULT_TABLE table=default.Table_1bb389d88a7c49ea86fc830f2393cb07 [id=b8ac486798e449c49221196462acbe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.576781 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc15abe1a4b46b693167c8fda7f9ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.581434 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d (DEFAULT_TABLE table=default.Table_eafc34b6a2f84ee48f05d17454df5bbb [id=936564561ccc4f92a831a90c6cd303fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.581617 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ab877bb53ad749da82c8b331ed9f4821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.581995 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d898f46beda343a781c0c7929dacde0d (DEFAULT_TABLE table=default.Table_c8c69b95a3b14bb496e64b147ad6ad8b [id=4abf76e263654cf38e0f9489cc11680a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:40.582233 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab877bb53ad749da82c8b331ed9f4821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.582813 1030 tablet_service.cc:1468] Processing CreateTablet for tablet eb1718d6cc4b4651b262020d5056b88e (DEFAULT_TABLE table=default.Table_045259525d4b454fbc351c409b78e67a [id=a1701c10e86f4b64b77cf90ce0ad6ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.583120 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0895c4127f0c408087c20ebfe5fb643e (DEFAULT_TABLE table=default.Table_a5c64e4e538143159ab5e58f3eb602af [id=a8e6fd4c910c48c7af5a2da738b2ff32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.583433 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 26aca80f5fef43fbbb911fd165b2cd05 (DEFAULT_TABLE table=default.Table_818abe262b4b41a4877229ae67d355b5 [id=717af2c308a247cdad9baa5e2869a63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.583585 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1faf79952c384f70b4fa38532dc9a7bc (DEFAULT_TABLE table=default.Table_f3e9de3fec8145649644fe7e4c9bfd7e [id=8793ec69a02c46d08e4e214895dbd207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.583721 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 37eebfb24ce84ac583a51529238b5fc1 (DEFAULT_TABLE table=default.Table_83d3b9777ac5429ab03a24bddd76bc36 [id=9aca89c6d0964ea8b487d1f525583211]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.583967 1011 tablet_service.cc:1468] Processing CreateTablet for tablet eff118a9b78a4badbc298d4cb93200c1 (DEFAULT_TABLE table=default.Table_dd0ca292d8b248ce8ddf004badf3ca25 [id=f7281cb6b77247b99e788b41b2069027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.584044 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2552170f354546b37f636182c8afeb (DEFAULT_TABLE table=default.Table_ea115540f06e4c1bac09bfd88eabda0a [id=12b953b4bee248a5bad30d9e2774486f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.584304 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3 (DEFAULT_TABLE table=default.Table_68221d261cec4a288a02a90ac62d03de [id=fd5cdd850e6a47a48c0623d71352fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.584718 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_c194642288ef486a9d9aac5a144c385e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.585260 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c194642288ef486a9d9aac5a144c385e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.588483 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f0bee21bf441d99fccdf823cb7be07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.589980 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_56030f59057b4ce39f60fb888dd0fc17" } new_table_name: "default.Table_da26dfc382ee4560bc995bf43cd00f1a" modify_external_catalogs: true
I20250625 07:52:40.590008 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f66d6700dcbf489286819ea4f80a75a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.590466 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f66d6700dcbf489286819ea4f80a75a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.591183 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e89d1c8b956c4d12a0b715252d470852" } modify_external_catalogs: true
I20250625 07:52:40.591320 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e89d1c8b956c4d12a0b715252d470852" } modify_external_catalogs: true
I20250625 07:52:40.592955 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b495ebc15ab14b7490f58c02290de736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.593421 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b495ebc15ab14b7490f58c02290de736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.597405 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca96d73ad9be44f29ea7fdfcf9132951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.597599 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aa89659db14725a910cc917ed9a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.597766 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f14b21749ab4b0b95a7662c190b2e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.597986 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b952ac910c4734bda9edd17efafda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.600832 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f925f53350b14044bff3f3f797641e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.603992 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0377a1e4099345e292541aeacd2d07ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.604131 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f6d472b3d1dd46fda16f6c09a2b641cd (DEFAULT_TABLE table=default.Table_08ff7050ded34087b0e7b538b53b1d3c [id=25bc4d873c734956bb238a505c0f6a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.604480 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d472b3d1dd46fda16f6c09a2b641cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.604846 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f99e19a3341cb90792caae491b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.610560 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc15abe1a4b46b693167c8fda7f9ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.610781 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d898f46beda343a781c0c7929dacde0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.614104 2475 raft_consensus.cc:491] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.614259 2475 raft_consensus.cc:513] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.614928 2475 leader_election.cc:290] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.615247 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1718d6cc4b4651b262020d5056b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.616899 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0895c4127f0c408087c20ebfe5fb643e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.620604 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26aca80f5fef43fbbb911fd165b2cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.621861 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 910981d1df5840969d32f93a5a6d5d4c (DEFAULT_TABLE table=default.Table_5ab142d37cc64674ba7522485ec90897 [id=05054f18f6ab4ae4a645702a938ecf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.622211 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910981d1df5840969d32f93a5a6d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.622534 1019 tablet_service.cc:1468] Processing CreateTablet for tablet f3ba15c29f7943fe9e83c33e76618f18 (DEFAULT_TABLE table=default.Table_4a2e696e0fa8443392bb829107de445e [id=7e7b2c0350ab44f4b1af0b0a12b80d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.622817 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ba15c29f7943fe9e83c33e76618f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.624891 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1faf79952c384f70b4fa38532dc9a7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.624944 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2f37ef18e9124e50bcfecf9631cef7ea (DEFAULT_TABLE table=default.Table_cc3b3771179f4d57b2d2a7e965e87550 [id=2f628dc3d26f4d61b83f767befe10b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.625254 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f37ef18e9124e50bcfecf9631cef7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.628024 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e92dc108e9fb4942a51cec78da1f0930 (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.628336 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92dc108e9fb4942a51cec78da1f0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.628758 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eebfb24ce84ac583a51529238b5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.630877 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e08746a1f3634a52b81c3ff1c628f0a7 (DEFAULT_TABLE table=default.Table_00bcc156be73401ca72064b265fc6c2c [id=8665f0662f5f411184fa46fc0db103e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.631178 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08746a1f3634a52b81c3ff1c628f0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.632877 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff118a9b78a4badbc298d4cb93200c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.632930 1012 tablet_service.cc:1468] Processing CreateTablet for tablet a432165859074eebaa2460ba3b5fba25 (DEFAULT_TABLE table=default.Table_afb2c447878f4e4ea10bbe04bcdaa8e6 [id=4a964c7e88eb4463a9ebf3847865aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.633244 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a432165859074eebaa2460ba3b5fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.636905 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2552170f354546b37f636182c8afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.637082 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f8a69991a59b4682aa1beddfe449a10d (DEFAULT_TABLE table=default.Table_413a830b8d974a5b96393e3559e2af92 [id=443fb20ed58a4d7384eb95fbe37737b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.637378 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a69991a59b4682aa1beddfe449a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.639014 1012 tablet_service.cc:1468] Processing CreateTablet for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846 (DEFAULT_TABLE table=default.Table_901005941b484ae7a758af04aa665f16 [id=f3846a19be9a4293916aadc7d4c7891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.639320 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.640900 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.640941 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe8718766841fd8188e4f358400a9e (DEFAULT_TABLE table=default.Table_1bf36482f7224ad9932b73d8ce0a5116 [id=b0d92cbd5a1440d0b20f398b554f4d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.641261 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe8718766841fd8188e4f358400a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.644723 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.645107 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f128431f6d0d4b2d8816ae169f9613d1 (DEFAULT_TABLE table=default.Table_6e61feb39d6145c0936cfa2316ddfdab [id=1b92149f05f74f99b64e1e608d5ea963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.645421 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f128431f6d0d4b2d8816ae169f9613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.646531 1018 tablet_service.cc:1468] Processing CreateTablet for tablet ae1300de7b364698935bdebdc6727c4a (DEFAULT_TABLE table=default.Table_42d926918dbf4b398f69aa2b3504f1b7 [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.646808 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1300de7b364698935bdebdc6727c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.650743 1015 tablet_service.cc:1468] Processing CreateTablet for tablet fb22dd02b6d44396a13710f6463f1b93 (DEFAULT_TABLE table=default.Table_ca9bb00cf9834f0797220f17c7ad930d [id=247137e23c204848ab7f3362db576401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.651069 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22dd02b6d44396a13710f6463f1b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.651813 2475 raft_consensus.cc:491] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.651945 2475 raft_consensus.cc:513] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.652150 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a642a958550a40a48453f47d01b062a2
I20250625 07:52:40.652494 2475 leader_election.cc:290] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:40.652842 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9bda35c7fae042e7af6631324add64d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.653409 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bda35c7fae042e7af6631324add64d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.654630 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a571983779854eb398b3b0b42c34f83a
I20250625 07:52:40.655225 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_fb8fd6ee8d994e9cabf8d6834fc42a1a" } new_table_name: "default.Table_d674965ce9e741bb881a96ae88af3859" modify_external_catalogs: true
I20250625 07:52:40.662830 2498 raft_consensus.cc:491] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.662987 2498 raft_consensus.cc:513] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.663507 2498 leader_election.cc:290] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
W20250625 07:52:40.529325 713 leader_election.cc:343] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a396bde4b5964697ad0090eab777ba0e
I20250625 07:52:40.531525 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4729a62b7434cc0bcd1eecbfe838faf" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.664547 986 leader_election.cc:304] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:40.664825 2661 raft_consensus.cc:2747] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:40.663966 713 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:45317, user_credentials={real_user=slave}} blocked reactor thread for 134662us
I20250625 07:52:40.666890 2498 consensus_queue.cc:1035] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:40.667460 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a642a958550a40a48453f47d01b062a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:40.669492 713 leader_election.cc:343] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.669605 713 leader_election.cc:304] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.669798 2498 raft_consensus.cc:2747] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:40.672551 2612 raft_consensus.cc:491] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.672772 2612 raft_consensus.cc:513] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.673409 2612 leader_election.cc:290] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.674926 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:40.675174 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 813f9f9fccc34685b73847c9d590ceb6 (table default.Table_6fbc60d9bf1c4a169d7565b1d870e351 [id=3f984673a2964bf19fdbb95a9da4e109]). Expected version 1 got 0
I20250625 07:52:40.675335 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:40.675549 596 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d7ed91509ba94562a9207c7d7c0d3e2b (table default.Table_ca90d59a48d947778cd899359e52ed41 [id=1353496ce19c4b14bdcc41400f3e9844]). Expected version 1 got 0
I20250625 07:52:40.678700 2407 ts_tablet_manager.cc:1918] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.678814 2407 log.cc:1199] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:40.679116 2407 ts_tablet_manager.cc:1939] T 8bc8f47541c54c4287ebb7b892daa1be P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.679477 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1412) took 2211 ms. Trace:
I20250625 07:52:40.679667 2475 raft_consensus.cc:491] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.679818 2475 raft_consensus.cc:513] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.679745 2407 rpcz_store.cc:276] 0625 07:52:38.467834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.467880 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:40.425135 (+1957255us) ts_tablet_manager.cc:1145] Deleting tablet 8bc8f47541c54c4287ebb7b892daa1be
0625 07:52:40.425145 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.498548 (+ 73403us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.678584 (+180036us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.679457 (+ 873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1957109}
I20250625 07:52:40.680351 2407 tablet_replica.cc:331] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.680429 2475 leader_election.cc:290] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:40.692365 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0] alter complete (version 3)
I20250625 07:52:40.695798 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_903b3fa5b06348ec8f2b92c77da28e7b" } new_table_name: "default.Table_5adb1c1991aa439fba28747ef698451d" modify_external_catalogs: true
I20250625 07:52:40.703980 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_cb77d581902c4391ac8bcdca79163105 [id=b1ec5aff86df4e9381d955d406326bde] alter complete (version 1)
W20250625 07:52:40.706246 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 199565us
I20250625 07:52:40.706436 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_73ebfb29d78146feb6bb0b4ae5633ebc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.708225 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73ebfb29d78146feb6bb0b4ae5633ebc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.588172 1158 raft_consensus.cc:357] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.672441 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f1d251171314118b66e3a921a4a50c5" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.717928 1158 raft_consensus.cc:383] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.718026 1158 raft_consensus.cc:738] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.718261 1158 consensus_queue.cc:260] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.718773 1158 ts_tablet_manager.cc:1428] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.190s user 0.001s sys 0.000s
I20250625 07:52:40.719045 1158 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.720376 1158 tablet_bootstrap.cc:654] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.674211 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a571983779854eb398b3b0b42c34f83a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.674865 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 60d2375fda2442cb9c566a2e640f0934 (DEFAULT_TABLE table=default.Table_d25be557c62b486592bc6f4c1025354e [id=28fb7ed6a3f348fbbf214380e3add79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.721035 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d2375fda2442cb9c566a2e640f0934. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:40.723664 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 46f81617a460458791e215b81c589241 (table default.Table_629ca82911b747e8af6c63294a5aae21 [id=044d275c5fbb468db72885400493f0ea], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46f81617a460458791e215b81c589241 consensus state has no leader
I20250625 07:52:40.725219 2475 raft_consensus.cc:491] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.725374 2475 raft_consensus.cc:513] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.725996 2475 leader_election.cc:290] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.732457 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1350) took 1583 ms. Trace:
I20250625 07:52:40.738451 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e06ecad6c7464ffe8ec6f632634307c9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.702392 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e06ecad6c7464ffe8ec6f632634307c9" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.702595 2498 raft_consensus.cc:491] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.741451 2498 raft_consensus.cc:513] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.742061 2498 leader_election.cc:290] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.706898 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c9aca996884ffeab88381dd3cb189b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.711010 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0a75906385744e5ba6938f451a2d7ec" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.726594 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e46f27344b848318dcd1e782819b75e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.743536 851 leader_election.cc:343] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:40.743803 851 leader_election.cc:343] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.743945 2407 raft_consensus.cc:2241] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.744213 2407 raft_consensus.cc:2270] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.744882 2407 ts_tablet_manager.cc:1905] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.745757 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47c9aca996884ffeab88381dd3cb189b" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:40.745193 851 leader_election.cc:343] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.750258 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0a75906385744e5ba6938f451a2d7ec" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.740806 1161 rpcz_store.cc:276] 0625 07:52:39.149436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.149583 (+ 147us) service_pool.cc:225] Handling call
0625 07:52:40.488512 (+1338929us) ts_tablet_manager.cc:1145] Deleting tablet f4bb74391a1c4e2792226d259a07f977
0625 07:52:40.488524 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.554079 (+ 65555us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.557084 (+ 3005us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.732428 (+175344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338751}
W20250625 07:52:40.751199 851 leader_election.cc:343] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.691013 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 21d11cdcccb54af1ab68b39f9f0474a0 (DEFAULT_TABLE table=default.Table_cfcdd655f66c4381b702e1d689f72291 [id=e31dd4d7a11f4318ad37825530b1f794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.750332 2475 raft_consensus.cc:491] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.753063 2475 raft_consensus.cc:513] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.753729 2475 leader_election.cc:290] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.753850 2475 raft_consensus.cc:491] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.753973 2475 raft_consensus.cc:513] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.754472 2475 leader_election.cc:290] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.754390 1161 tablet_replica.cc:331] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:40.755722 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0102a3b176e146a8b19abe22d3aba656" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.756108 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35a93e9198d1417bb1b8e00560921483" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.756898 851 leader_election.cc:343] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:40.757337 851 leader_election.cc:343] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.759399 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d11cdcccb54af1ab68b39f9f0474a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.760342 1161 raft_consensus.cc:2241] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.764956 1161 raft_consensus.cc:2270] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.771667 1158 tablet_bootstrap.cc:492] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:40.774904 1158 ts_tablet_manager.cc:1397] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.000s
I20250625 07:52:40.778185 2407 ts_tablet_manager.cc:1918] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.780391 1161 ts_tablet_manager.cc:1905] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.789065 2475 raft_consensus.cc:491] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.789247 2475 raft_consensus.cc:513] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.789988 2475 leader_election.cc:290] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:40.792640 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.796815 2407 log.cc:1199] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:40.797289 2407 ts_tablet_manager.cc:1939] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.796295 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet f4bb74391a1c4e2792226d259a07f977 (table default.Table_26efe610b5ee4ac1b55ac4d7d91c7040 [id=ba3a96b6e2ae4c2488ee0b91816af1f7]) successfully deleted
W20250625 07:52:40.797443 851 leader_election.cc:343] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.801419 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1413) took 2333 ms. Trace:
I20250625 07:52:40.802273 2407 rpcz_store.cc:276] 0625 07:52:38.468160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.468204 (+ 44us) service_pool.cc:225] Handling call
0625 07:52:40.680300 (+2212096us) ts_tablet_manager.cc:1145] Deleting tablet 9f1d251171314118b66e3a921a4a50c5
0625 07:52:40.680310 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.765918 (+ 85608us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.778051 (+ 12133us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.801394 (+ 23343us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2841,"tablet-delete.queue_time_us":2211950}
I20250625 07:52:40.803063 2407 tablet_replica.cc:331] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.803572 2407 raft_consensus.cc:2241] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.804059 2407 raft_consensus.cc:2270] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.804939 2407 ts_tablet_manager.cc:1905] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.803313 1158 raft_consensus.cc:357] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.810717 1158 raft_consensus.cc:383] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.813129 1158 raft_consensus.cc:738] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.816432 2475 raft_consensus.cc:491] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.816610 2475 raft_consensus.cc:513] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.816988 1158 consensus_queue.cc:260] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:40.817210 2475 leader_election.cc:290] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.817536 1158 ts_tablet_manager.cc:1428] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.039s user 0.004s sys 0.000s
I20250625 07:52:40.817831 1158 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:40.817866 888 tablet_service.cc:1468] Processing CreateTablet for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d (DEFAULT_TABLE table=default.Table_eafc34b6a2f84ee48f05d17454df5bbb [id=936564561ccc4f92a831a90c6cd303fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.818285 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.819191 1158 tablet_bootstrap.cc:654] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.819100 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9578d77db00242d48f4e8f6f8263ef09" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:40.826716 851 leader_election.cc:343] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.822749 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_45dacbb132754fd98560d96145a5030f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.827271 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8f7140aa42f54e6b80e1e6ce1202eb2c" } new_table_name: "default.Table_f2a6e24fe227418fbfba468a5d568c2a" modify_external_catalogs: true
W20250625 07:52:40.827657 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45dacbb132754fd98560d96145a5030f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.823514 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9e73032861a4dc28832fdcc1707bfe0 consensus state has no leader
I20250625 07:52:40.825114 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c24353dd31c0489299840521478b82f4" } new_table_name: "default.Table_eb5b85131bf7494c854558bb7610441b" modify_external_catalogs: true
I20250625 07:52:40.825433 595 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 483e104c78d742869fbe2a51460ff100
I20250625 07:52:40.831246 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ea9b2b65ce2b451993172b74d520f24c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:40.831466 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cba2a3adedc14eb1a572c3f07e0709c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.831849 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea9b2b65ce2b451993172b74d520f24c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.832022 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba2a3adedc14eb1a572c3f07e0709c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:40.825646 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0f54f7d36a74105a2d2215682a5f3b4 consensus state has no leader
I20250625 07:52:40.826699 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_19f5f9aa6d8545839a1b66329c73df04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:40.833686 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19f5f9aa6d8545839a1b66329c73df04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:40.835700 597 master_service.cc:948] ReplaceTablet: received request to replace tablet e15c7ac8ff934a8ca40ac480c7c352ab from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:40.841328 592 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 4da889781dbd41aebac86e9833cf2b7e (table default.Table_c4ec962d89484a528ecd0f46a281c91b [id=a3fe36da128741949f4beb0021a3bcba]). Expected version 1 got 0
I20250625 07:52:40.842509 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_381f2b07c1fe4a47adbf6b5f45f738b5" } modify_external_catalogs: true
I20250625 07:52:40.842665 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_381f2b07c1fe4a47adbf6b5f45f738b5" } modify_external_catalogs: true
I20250625 07:52:40.855543 1160 ts_tablet_manager.cc:1918] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.855669 1160 log.cc:1199] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9f31104720614b028e373ae1500c31cc
W20250625 07:52:40.857686 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dde275b2d99a4b0cb008a2fba9945960 consensus state has no leader
I20250625 07:52:40.860209 886 tablet_service.cc:1468] Processing CreateTablet for tablet d898f46beda343a781c0c7929dacde0d (DEFAULT_TABLE table=default.Table_c8c69b95a3b14bb496e64b147ad6ad8b [id=4abf76e263654cf38e0f9489cc11680a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.860769 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d898f46beda343a781c0c7929dacde0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.862167 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 173a565916c1440493ece348599ddba3 (table default.Table_cb77d581902c4391ac8bcdca79163105 [id=b1ec5aff86df4e9381d955d406326bde]). Expected version 1 got 0
I20250625 07:52:40.864495 591 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3638af2256ed45d68ee8334b9b3c30b4" } modify_external_catalogs: true
I20250625 07:52:40.864787 591 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3638af2256ed45d68ee8334b9b3c30b4" } modify_external_catalogs: true
W20250625 07:52:40.871206 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20d6334831b942399273d8d1314d7908 consensus state has no leader
I20250625 07:52:40.871692 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_656035a4978b49bd86d2e4462a67a4dc" } modify_external_catalogs: true
I20250625 07:52:40.871740 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6b33905fc1b444d6876011512196d67b" } modify_external_catalogs: true
I20250625 07:52:40.871902 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_6b33905fc1b444d6876011512196d67b" } modify_external_catalogs: true
I20250625 07:52:40.871882 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_656035a4978b49bd86d2e4462a67a4dc" } modify_external_catalogs: true
I20250625 07:52:40.874076 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fd6c3a490883477dbafcb74d39be7345
I20250625 07:52:40.874621 2721 raft_consensus.cc:491] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:40.875037 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7863b48536e14b2a8d9173c8aa787fbf consensus state has no leader
I20250625 07:52:40.874975 2721 raft_consensus.cc:513] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.875463 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e15c7ac8ff934a8ca40ac480c7c352ab
I20250625 07:52:40.875890 597 catalog_manager.cc:6652] ReplaceTablet: tablet e15c7ac8ff934a8ca40ac480c7c352ab deleted and replaced by tablet 1021a582ade04f80ba8e582e5040a217
I20250625 07:52:40.875931 2721 leader_election.cc:290] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.877266 891 tablet_service.cc:1468] Processing CreateTablet for tablet eb1718d6cc4b4651b262020d5056b88e (DEFAULT_TABLE table=default.Table_045259525d4b454fbc351c409b78e67a [id=a1701c10e86f4b64b77cf90ce0ad6ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.877869 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1718d6cc4b4651b262020d5056b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.881178 2407 ts_tablet_manager.cc:1918] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.881654 2407 log.cc:1199] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:40.881984 2407 ts_tablet_manager.cc:1939] T 70094e6c7e644f62bf5ef04dcfb4c380 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:40.882754 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1414) took 2414 ms. Trace:
I20250625 07:52:40.882902 2407 rpcz_store.cc:276] 0625 07:52:38.468472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.468520 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:40.803002 (+2334482us) ts_tablet_manager.cc:1145] Deleting tablet 70094e6c7e644f62bf5ef04dcfb4c380
0625 07:52:40.803016 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.877164 (+ 74148us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.881067 (+ 3903us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.882735 (+ 1668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2334330}
I20250625 07:52:40.883409 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1406) took 2417 ms. Trace:
I20250625 07:52:40.883530 2407 rpcz_store.cc:276] 0625 07:52:38.466050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.468750 (+ 2700us) service_pool.cc:225] Handling call
0625 07:52:40.883294 (+2414544us) ts_tablet_manager.cc:1145] Deleting tablet 53bf5c84b12d4bb399c16eb6e77a28ef
0625 07:52:40.883303 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.883397 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2414333}
I20250625 07:52:40.883828 2407 tablet_replica.cc:331] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:40.884073 2407 raft_consensus.cc:2241] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:40.884405 2407 pending_rounds.cc:70] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:40.884507 2407 pending_rounds.cc:77] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432272468164608 op_type: NO_OP noop_request { }
I20250625 07:52:40.884667 2407 raft_consensus.cc:2887] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:40.884825 2407 raft_consensus.cc:2270] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.885385 2407 ts_tablet_manager.cc:1905] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.881570 1162 tablet_bootstrap.cc:492] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:40.887197 1162 tablet_bootstrap.cc:654] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.889837 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab with delete_type TABLET_DATA_DELETED (replaced by tablet 1021a582ade04f80ba8e582e5040a217) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.890132 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6c3a490883477dbafcb74d39be7345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.890880 885 tablet_service.cc:1468] Processing CreateTablet for tablet 26aca80f5fef43fbbb911fd165b2cd05 (DEFAULT_TABLE table=default.Table_818abe262b4b41a4877229ae67d355b5 [id=717af2c308a247cdad9baa5e2869a63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.891389 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26aca80f5fef43fbbb911fd165b2cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.894754 2475 raft_consensus.cc:491] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.894910 2475 raft_consensus.cc:513] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.895536 2475 leader_election.cc:290] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:40.896373 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506794253c84486faa926de885c48c58" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:40.896763 749 tablet_service.cc:1468] Processing CreateTablet for tablet ed4166d5cece4934a0dfb0c9cf3cc690 (DEFAULT_TABLE table=default.Table_880f1c0b45ed4ab38c040de1a705f41c [id=422648aa29d3412f9b1653ba53b95324]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:40.896978 851 leader_election.cc:343] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.897157 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed4166d5cece4934a0dfb0c9cf3cc690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.897997 891 tablet_service.cc:1468] Processing CreateTablet for tablet 0895c4127f0c408087c20ebfe5fb643e (DEFAULT_TABLE table=default.Table_a5c64e4e538143159ab5e58f3eb602af [id=a8e6fd4c910c48c7af5a2da738b2ff32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.898363 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0895c4127f0c408087c20ebfe5fb643e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.904095 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bbf1cfac4e14a20b3fb53607cdf100f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.904681 752 tablet_service.cc:1468] Processing CreateTablet for tablet ca96d73ad9be44f29ea7fdfcf9132951 (DEFAULT_TABLE table=default.Table_08fde6e3798a4aa0ab2006ff6532dd63 [id=8aa99a74daeb4d58a948d8e78e4cc709]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.905115 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca96d73ad9be44f29ea7fdfcf9132951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.907313 888 tablet_service.cc:1468] Processing CreateTablet for tablet 1faf79952c384f70b4fa38532dc9a7bc (DEFAULT_TABLE table=default.Table_f3e9de3fec8145649644fe7e4c9bfd7e [id=8793ec69a02c46d08e4e214895dbd207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.907684 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1faf79952c384f70b4fa38532dc9a7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.910451 742 tablet_service.cc:1468] Processing CreateTablet for tablet 839f99e19a3341cb90792caae491b3b4 (DEFAULT_TABLE table=default.Table_a59eca497c7e423b801c1f7e4e1ed131 [id=fde83281868e41bbbf7df57ba3225be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.910845 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f99e19a3341cb90792caae491b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.913453 1163 tablet_bootstrap.cc:492] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.913671 1163 ts_tablet_manager.cc:1397] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.393s user 0.001s sys 0.005s
I20250625 07:52:40.917063 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6b4bca32de4a538071a652409e93b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:40.917521 739 tablet_service.cc:1468] Processing CreateTablet for tablet f925f53350b14044bff3f3f797641e4e (DEFAULT_TABLE table=default.Table_5bcfcafad68c45a28e30214ffb855371 [id=7966682bd34146818413026936a049ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.917882 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f925f53350b14044bff3f3f797641e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.920033 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0377a1e4099345e292541aeacd2d07ff (DEFAULT_TABLE table=default.Table_114f4c0ed74b446b9ddf211e56589e6b [id=e8351eff387846b4b1a094ba851f724e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.920364 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0377a1e4099345e292541aeacd2d07ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.921247 758 tablet_service.cc:1468] Processing CreateTablet for tablet 05b952ac910c4734bda9edd17efafda0 (DEFAULT_TABLE table=default.Table_87be1efdccf045a9877e5f62907de1f6 [id=98ba61c1ea47423982578ed6ae3f7df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.921545 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05b952ac910c4734bda9edd17efafda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.926437 749 tablet_service.cc:1468] Processing CreateTablet for tablet 47f0bee21bf441d99fccdf823cb7be07 (DEFAULT_TABLE table=default.Table_4970ea915b7042629008e088f9d45013 [id=21b03aeae46d444b9e561294fa86e3da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.926774 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47f0bee21bf441d99fccdf823cb7be07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.929407 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1faf79952c384f70b4fa38532dc9a7bc (DEFAULT_TABLE table=default.Table_f3e9de3fec8145649644fe7e4c9bfd7e [id=8793ec69a02c46d08e4e214895dbd207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.929716 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1faf79952c384f70b4fa38532dc9a7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.931350 739 tablet_service.cc:1468] Processing CreateTablet for tablet 82aa89659db14725a910cc917ed9a7f9 (DEFAULT_TABLE table=default.Table_7d8f8cc6cbff4a34b036a097f4fa40ce [id=bdc819b1871c4e82b290288da6b2eaa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.931640 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aa89659db14725a910cc917ed9a7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.936102 758 tablet_service.cc:1468] Processing CreateTablet for tablet eff118a9b78a4badbc298d4cb93200c1 (DEFAULT_TABLE table=default.Table_dd0ca292d8b248ce8ddf004badf3ca25 [id=f7281cb6b77247b99e788b41b2069027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.936162 747 tablet_service.cc:1468] Processing CreateTablet for tablet eb1718d6cc4b4651b262020d5056b88e (DEFAULT_TABLE table=default.Table_045259525d4b454fbc351c409b78e67a [id=a1701c10e86f4b64b77cf90ce0ad6ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.936589 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb1718d6cc4b4651b262020d5056b88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.936614 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0895c4127f0c408087c20ebfe5fb643e (DEFAULT_TABLE table=default.Table_a5c64e4e538143159ab5e58f3eb602af [id=a8e6fd4c910c48c7af5a2da738b2ff32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.936988 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0895c4127f0c408087c20ebfe5fb643e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.942016 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff118a9b78a4badbc298d4cb93200c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.943291 745 tablet_service.cc:1468] Processing CreateTablet for tablet 37eebfb24ce84ac583a51529238b5fc1 (DEFAULT_TABLE table=default.Table_83d3b9777ac5429ab03a24bddd76bc36 [id=9aca89c6d0964ea8b487d1f525583211]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.943620 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eebfb24ce84ac583a51529238b5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.945540 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a30941e3b9c475db17f961c38d96ef4 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:40.946902 1163 raft_consensus.cc:357] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.947178 1163 raft_consensus.cc:383] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.947286 1163 raft_consensus.cc:738] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.947528 1163 consensus_queue.cc:260] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.948086 1163 ts_tablet_manager.cc:1428] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20250625 07:52:40.948405 1163 tablet_bootstrap.cc:492] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:40.949647 748 tablet_service.cc:1468] Processing CreateTablet for tablet d898f46beda343a781c0c7929dacde0d (DEFAULT_TABLE table=default.Table_c8c69b95a3b14bb496e64b147ad6ad8b [id=4abf76e263654cf38e0f9489cc11680a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.950040 1163 tablet_bootstrap.cc:654] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:40.950075 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d898f46beda343a781c0c7929dacde0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.951574 2612 raft_consensus.cc:491] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.951701 2612 raft_consensus.cc:513] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.952288 2612 leader_election.cc:290] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.956450 755 tablet_service.cc:1468] Processing CreateTablet for tablet edc15abe1a4b46b693167c8fda7f9ce1 (DEFAULT_TABLE table=default.Table_1bb389d88a7c49ea86fc830f2393cb07 [id=b8ac486798e449c49221196462acbe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.956878 886 tablet_service.cc:1468] Processing CreateTablet for tablet eff118a9b78a4badbc298d4cb93200c1 (DEFAULT_TABLE table=default.Table_dd0ca292d8b248ce8ddf004badf3ca25 [id=f7281cb6b77247b99e788b41b2069027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.957185 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc15abe1a4b46b693167c8fda7f9ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.957353 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff118a9b78a4badbc298d4cb93200c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.960985 753 tablet_service.cc:1468] Processing CreateTablet for tablet 26aca80f5fef43fbbb911fd165b2cd05 (DEFAULT_TABLE table=default.Table_818abe262b4b41a4877229ae67d355b5 [id=717af2c308a247cdad9baa5e2869a63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.961405 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26aca80f5fef43fbbb911fd165b2cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.962138 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet da895c90d6ee40379fe8181c2dbd7a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.963183 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2552170f354546b37f636182c8afeb (DEFAULT_TABLE table=default.Table_ea115540f06e4c1bac09bfd88eabda0a [id=12b953b4bee248a5bad30d9e2774486f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.963223 883 tablet_service.cc:1468] Processing CreateTablet for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3 (DEFAULT_TABLE table=default.Table_68221d261cec4a288a02a90ac62d03de [id=fd5cdd850e6a47a48c0623d71352fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.963549 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2552170f354546b37f636182c8afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.963726 878 tablet_service.cc:1468] Processing CreateTablet for tablet f6d472b3d1dd46fda16f6c09a2b641cd (DEFAULT_TABLE table=default.Table_08ff7050ded34087b0e7b538b53b1d3c [id=25bc4d873c734956bb238a505c0f6a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.964082 893 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe8718766841fd8188e4f358400a9e (DEFAULT_TABLE table=default.Table_1bf36482f7224ad9932b73d8ce0a5116 [id=b0d92cbd5a1440d0b20f398b554f4d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.964419 889 tablet_service.cc:1468] Processing CreateTablet for tablet f8a69991a59b4682aa1beddfe449a10d (DEFAULT_TABLE table=default.Table_413a830b8d974a5b96393e3559e2af92 [id=443fb20ed58a4d7384eb95fbe37737b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.964783 895 tablet_service.cc:1468] Processing CreateTablet for tablet e08746a1f3634a52b81c3ff1c628f0a7 (DEFAULT_TABLE table=default.Table_00bcc156be73401ca72064b265fc6c2c [id=8665f0662f5f411184fa46fc0db103e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.964862 876 tablet_service.cc:1468] Processing CreateTablet for tablet f3ba15c29f7943fe9e83c33e76618f18 (DEFAULT_TABLE table=default.Table_4a2e696e0fa8443392bb829107de445e [id=7e7b2c0350ab44f4b1af0b0a12b80d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.965180 880 tablet_service.cc:1468] Processing CreateTablet for tablet a432165859074eebaa2460ba3b5fba25 (DEFAULT_TABLE table=default.Table_afb2c447878f4e4ea10bbe04bcdaa8e6 [id=4a964c7e88eb4463a9ebf3847865aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.965421 890 tablet_service.cc:1468] Processing CreateTablet for tablet f128431f6d0d4b2d8816ae169f9613d1 (DEFAULT_TABLE table=default.Table_6e61feb39d6145c0936cfa2316ddfdab [id=1b92149f05f74f99b64e1e608d5ea963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.965473 877 tablet_service.cc:1468] Processing CreateTablet for tablet e92dc108e9fb4942a51cec78da1f0930 (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.965473 884 tablet_service.cc:1468] Processing CreateTablet for tablet ae1300de7b364698935bdebdc6727c4a (DEFAULT_TABLE table=default.Table_42d926918dbf4b398f69aa2b3504f1b7 [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.965852 894 tablet_service.cc:1468] Processing CreateTablet for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846 (DEFAULT_TABLE table=default.Table_901005941b484ae7a758af04aa665f16 [id=f3846a19be9a4293916aadc7d4c7891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.966065 882 tablet_service.cc:1468] Processing CreateTablet for tablet 910981d1df5840969d32f93a5a6d5d4c (DEFAULT_TABLE table=default.Table_5ab142d37cc64674ba7522485ec90897 [id=05054f18f6ab4ae4a645702a938ecf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.966924 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e46f27344b848318dcd1e782819b75e" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.967069 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0102a3b176e146a8b19abe22d3aba656" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.967408 885 tablet_service.cc:1468] Processing CreateTablet for tablet 37eebfb24ce84ac583a51529238b5fc1 (DEFAULT_TABLE table=default.Table_83d3b9777ac5429ab03a24bddd76bc36 [id=9aca89c6d0964ea8b487d1f525583211]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.967999 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35a93e9198d1417bb1b8e00560921483" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.968016 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.968946 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2f37ef18e9124e50bcfecf9631cef7ea (DEFAULT_TABLE table=default.Table_cc3b3771179f4d57b2d2a7e965e87550 [id=2f628dc3d26f4d61b83f767befe10b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.969297 881 tablet_service.cc:1468] Processing CreateTablet for tablet fb22dd02b6d44396a13710f6463f1b93 (DEFAULT_TABLE table=default.Table_ca9bb00cf9834f0797220f17c7ad930d [id=247137e23c204848ab7f3362db576401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.969587 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506794253c84486faa926de885c48c58" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.969620 887 tablet_service.cc:1468] Processing CreateTablet for tablet 60d2375fda2442cb9c566a2e640f0934 (DEFAULT_TABLE table=default.Table_d25be557c62b486592bc6f4c1025354e [id=28fb7ed6a3f348fbbf214380e3add79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.969847 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9578d77db00242d48f4e8f6f8263ef09" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:40.973066 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d472b3d1dd46fda16f6c09a2b641cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.974195 1160 ts_tablet_manager.cc:1939] T 9f31104720614b028e373ae1500c31cc P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:40.974962 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1394) took 1351 ms. Trace:
I20250625 07:52:40.975042 1160 rpcz_store.cc:276] 0625 07:52:39.623755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.623806 (+ 51us) service_pool.cc:225] Handling call
0625 07:52:40.511425 (+887619us) ts_tablet_manager.cc:1145] Deleting tablet 9f31104720614b028e373ae1500c31cc
0625 07:52:40.511434 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.547151 (+ 35717us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.855430 (+308279us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.974939 (+119509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":887377}
I20250625 07:52:40.975406 1160 tablet_replica.cc:331] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:40.975584 1160 raft_consensus.cc:2241] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:40.975726 1160 raft_consensus.cc:2270] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:40.976104 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe8718766841fd8188e4f358400a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.976246 1160 ts_tablet_manager.cc:1905] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:40.978778 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet d43d98c0c51b40c6bdb0c2705465038e
I20250625 07:52:40.979544 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 670974e43d7e4886bfe5eb3cf36ab6ce
I20250625 07:52:40.980931 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a69991a59b4682aa1beddfe449a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.982939 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.983163 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d43d98c0c51b40c6bdb0c2705465038e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:40.984933 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08746a1f3634a52b81c3ff1c628f0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.986901 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a642a958550a40a48453f47d01b062a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.987052 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a571983779854eb398b3b0b42c34f83a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:40.987344 891 tablet_service.cc:1468] Processing CreateTablet for tablet 21d11cdcccb54af1ab68b39f9f0474a0 (DEFAULT_TABLE table=default.Table_cfcdd655f66c4381b702e1d689f72291 [id=e31dd4d7a11f4318ad37825530b1f794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:40.987517 2612 raft_consensus.cc:491] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:40.987653 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d11cdcccb54af1ab68b39f9f0474a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.987651 2612 raft_consensus.cc:513] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.988301 2612 leader_election.cc:290] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:40.989892 1161 ts_tablet_manager.cc:1918] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:40.989998 1161 log.cc:1199] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0bfcef152c9844a59989c532bd60c07d
I20250625 07:52:40.990265 1161 ts_tablet_manager.cc:1939] T 0bfcef152c9844a59989c532bd60c07d P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:40.991302 1163 tablet_bootstrap.cc:492] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:40.991451 1163 ts_tablet_manager.cc:1397] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250625 07:52:40.992590 1163 raft_consensus.cc:357] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.992799 1163 raft_consensus.cc:383] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:40.992847 1163 raft_consensus.cc:738] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:40.992942 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ba15c29f7943fe9e83c33e76618f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.993021 1163 consensus_queue.cc:260] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:40.994057 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a432165859074eebaa2460ba3b5fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.996049 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f128431f6d0d4b2d8816ae169f9613d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.997115 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1300de7b364698935bdebdc6727c4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:40.968972 850 leader_election.cc:343] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:40.997306 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.997344 850 leader_election.cc:304] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:40.997907 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910981d1df5840969d32f93a5a6d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.998659 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92dc108e9fb4942a51cec78da1f0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.999527 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eebfb24ce84ac583a51529238b5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:40.999992 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f37ef18e9124e50bcfecf9631cef7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.000569 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22dd02b6d44396a13710f6463f1b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.000914 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1351) took 1788 ms. Trace:
I20250625 07:52:41.000970 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.001024 1161 rpcz_store.cc:276] 0625 07:52:39.212099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.212152 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:40.752969 (+1540817us) ts_tablet_manager.cc:1145] Deleting tablet 0bfcef152c9844a59989c532bd60c07d
0625 07:52:40.753308 (+ 339us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:40.850423 (+ 97115us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:40.989788 (+139365us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.000888 (+ 11100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540205}
I20250625 07:52:41.001498 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0bfcef152c9844a59989c532bd60c07d (table default.Table_1f8502fb577748268f87ac63e15ec4f5 [id=b6cfc505ed734958966b9e75b0e39415]) successfully deleted
I20250625 07:52:41.001835 1161 tablet_replica.cc:331] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.002035 1161 raft_consensus.cc:2241] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.002203 1161 raft_consensus.cc:2270] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.002328 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 0102a3b176e146a8b19abe22d3aba656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.002502 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d2375fda2442cb9c566a2e640f0934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.002545 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e46f27344b848318dcd1e782819b75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.002697 888 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6c3a490883477dbafcb74d39be7345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.002733 1161 ts_tablet_manager.cc:1905] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.002794 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab with delete_type TABLET_DATA_DELETED (replaced by tablet 1021a582ade04f80ba8e582e5040a217) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.006980 742 tablet_service.cc:1468] Processing CreateTablet for tablet 6f14b21749ab4b0b95a7662c190b2e62 (DEFAULT_TABLE table=default.Table_330c8b7943384fccb11c9490fd7dd310 [id=ff627b4ecef449f39f73038d27da68b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.007124 2475 raft_consensus.cc:2747] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.007462 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f14b21749ab4b0b95a7662c190b2e62. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.007589 850 leader_election.cc:343] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.007755 850 leader_election.cc:304] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.008003 2475 raft_consensus.cc:2747] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.008596 850 leader_election.cc:343] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.008764 850 leader_election.cc:304] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.008986 2475 raft_consensus.cc:2747] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.009361 850 leader_election.cc:343] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.009508 850 leader_election.cc:304] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.009718 2475 raft_consensus.cc:2747] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.009681 752 tablet_service.cc:1468] Processing CreateTablet for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d (DEFAULT_TABLE table=default.Table_eafc34b6a2f84ee48f05d17454df5bbb [id=936564561ccc4f92a831a90c6cd303fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.009953 1163 ts_tablet_manager.cc:1428] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250625 07:52:41.010043 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a33bc6e1f244c7bd7bc2cde5b5160d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.010177 850 leader_election.cc:343] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.010324 850 leader_election.cc:304] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.010527 2475 raft_consensus.cc:2747] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.010871 850 leader_election.cc:343] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.010994 850 leader_election.cc:304] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.011200 2475 raft_consensus.cc:2747] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.011521 850 leader_election.cc:343] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.011651 850 leader_election.cc:304] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.011897 2475 raft_consensus.cc:2747] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.010257 1163 tablet_bootstrap.cc:492] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:41.013643 850 leader_election.cc:343] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.013800 850 leader_election.cc:304] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.014046 2475 raft_consensus.cc:2747] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.014389 850 leader_election.cc:343] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.015679 850 leader_election.cc:304] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.015954 2475 raft_consensus.cc:2747] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.018986 745 tablet_service.cc:1468] Processing CreateTablet for tablet f6d472b3d1dd46fda16f6c09a2b641cd (DEFAULT_TABLE table=default.Table_08ff7050ded34087b0e7b538b53b1d3c [id=25bc4d873c734956bb238a505c0f6a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.019380 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d472b3d1dd46fda16f6c09a2b641cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.018987 747 tablet_service.cc:1468] Processing CreateTablet for tablet 37fe8718766841fd8188e4f358400a9e (DEFAULT_TABLE table=default.Table_1bf36482f7224ad9932b73d8ce0a5116 [id=b0d92cbd5a1440d0b20f398b554f4d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.019675 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37fe8718766841fd8188e4f358400a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.019865 750 tablet_service.cc:1468] Processing CreateTablet for tablet f8a69991a59b4682aa1beddfe449a10d (DEFAULT_TABLE table=default.Table_413a830b8d974a5b96393e3559e2af92 [id=443fb20ed58a4d7384eb95fbe37737b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.020176 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a69991a59b4682aa1beddfe449a10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.021987 1161 ts_tablet_manager.cc:1918] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.022097 1161 log.cc:1199] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/cf175e17865e40ceb26aa955b91c2c13
I20250625 07:52:41.022433 1161 ts_tablet_manager.cc:1939] T cf175e17865e40ceb26aa955b91c2c13 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.022856 2661 raft_consensus.cc:491] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.024148 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2f37ef18e9124e50bcfecf9631cef7ea (DEFAULT_TABLE table=default.Table_cc3b3771179f4d57b2d2a7e965e87550 [id=2f628dc3d26f4d61b83f767befe10b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.024511 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f37ef18e9124e50bcfecf9631cef7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.023047 2661 raft_consensus.cc:513] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.026278 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7afdcd9db0394301b8a6504a6adf9b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.026445 746 tablet_service.cc:1468] Processing CreateTablet for tablet 910981d1df5840969d32f93a5a6d5d4c (DEFAULT_TABLE table=default.Table_5ab142d37cc64674ba7522485ec90897 [id=05054f18f6ab4ae4a645702a938ecf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.026769 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910981d1df5840969d32f93a5a6d5d4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.026993 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7afdcd9db0394301b8a6504a6adf9b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.029095 751 tablet_service.cc:1468] Processing CreateTablet for tablet f3ba15c29f7943fe9e83c33e76618f18 (DEFAULT_TABLE table=default.Table_4a2e696e0fa8443392bb829107de445e [id=7e7b2c0350ab44f4b1af0b0a12b80d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.029434 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ba15c29f7943fe9e83c33e76618f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.030438 757 tablet_service.cc:1468] Processing CreateTablet for tablet e08746a1f3634a52b81c3ff1c628f0a7 (DEFAULT_TABLE table=default.Table_00bcc156be73401ca72064b265fc6c2c [id=8665f0662f5f411184fa46fc0db103e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.030752 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08746a1f3634a52b81c3ff1c628f0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.033015 743 tablet_service.cc:1468] Processing CreateTablet for tablet a432165859074eebaa2460ba3b5fba25 (DEFAULT_TABLE table=default.Table_afb2c447878f4e4ea10bbe04bcdaa8e6 [id=4a964c7e88eb4463a9ebf3847865aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.033382 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a432165859074eebaa2460ba3b5fba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.034946 2661 leader_election.cc:290] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.036041 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bb21510ec96d482f82b38d009b3d0713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.036085 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1c20f484893f410e84bb744f4ac802a6" } new_table_name: "default.Table_aa37d818be3b4a62ba68b85e674b01ab" modify_external_catalogs: true
W20250625 07:52:41.036594 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb21510ec96d482f82b38d009b3d0713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.036998 756 tablet_service.cc:1468] Processing CreateTablet for tablet e92dc108e9fb4942a51cec78da1f0930 (DEFAULT_TABLE table=default.Table_b374bdf422d9423b85441edf89cc3c95 [id=1f59072dca49439c8de53b0f89a6791e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.037351 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e92dc108e9fb4942a51cec78da1f0930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.038837 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aaa21f170574b79ae5d89ff2cc423ad" } modify_external_catalogs: true
I20250625 07:52:41.038997 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aaa21f170574b79ae5d89ff2cc423ad" } modify_external_catalogs: true
I20250625 07:52:41.040912 1158 tablet_bootstrap.cc:492] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:41.041074 1158 ts_tablet_manager.cc:1397] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.223s user 0.006s sys 0.000s
I20250625 07:52:41.041046 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e84a089995ef4ad78eb9025893a5683d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.041548 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e84a089995ef4ad78eb9025893a5683d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.041857 1158 raft_consensus.cc:357] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.042032 1158 raft_consensus.cc:383] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.042086 1158 raft_consensus.cc:738] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.042366 1158 consensus_queue.cc:260] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.042860 1158 ts_tablet_manager.cc:1428] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:41.042852 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_00a5bff9b246451ca66616d3b60fed60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.043164 1158 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:41.043332 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00a5bff9b246451ca66616d3b60fed60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.044682 744 tablet_service.cc:1468] Processing CreateTablet for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846 (DEFAULT_TABLE table=default.Table_901005941b484ae7a758af04aa665f16 [id=f3846a19be9a4293916aadc7d4c7891a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.045101 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee2ae1eb4b0b449c8f7e1dae58dd3846. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.047061 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9f31104720614b028e373ae1500c31cc (table default.Table_7c4ed6c1f8c0466e9a606dc7aeaea289 [id=82f3c359f3be4716987d150b7251db98]) successfully deleted
I20250625 07:52:41.047843 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_50f0c18132ce48cf8b4242ae079512e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.048349 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50f0c18132ce48cf8b4242ae079512e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.049461 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2e1e160bec8647f2b41862f84d3e4817" } new_table_name: "default.Table_06f992952f9f4dc6b998e2be5694d564" modify_external_catalogs: true
I20250625 07:52:41.050637 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.051841 879 tablet_service.cc:1515] Processing DeleteTablet for tablet d43d98c0c51b40c6bdb0c2705465038e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.052057 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4f2552170f354546b37f636182c8afeb (DEFAULT_TABLE table=default.Table_ea115540f06e4c1bac09bfd88eabda0a [id=12b953b4bee248a5bad30d9e2774486f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.052217 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.053047 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f2552170f354546b37f636182c8afeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.056533 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4da889781dbd41aebac86e9833cf2b7e" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:41.057468 985 leader_election.cc:304] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.058609 2723 raft_consensus.cc:491] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.058737 2723 raft_consensus.cc:513] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.059316 2723 leader_election.cc:290] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:41.059521 2612 raft_consensus.cc:491] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.059633 2612 raft_consensus.cc:513] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.060165 2612 leader_election.cc:290] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:41.060783 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f9f9fccc34685b73847c9d590ceb6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.061439 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 9b83f4c8544946d9adb3346df200b215
I20250625 07:52:41.062320 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b83f4c8544946d9adb3346df200b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:41.062551 851 leader_election.cc:343] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.062752 2661 raft_consensus.cc:2747] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.064795 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1bb389d88a7c49ea86fc830f2393cb07" } new_table_name: "default.Table_58575f81b4464844b62a265bb1f5b4e3" modify_external_catalogs: true
I20250625 07:52:41.068159 2661 raft_consensus.cc:491] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.068321 2661 raft_consensus.cc:513] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.070291 2661 leader_election.cc:290] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.071727 986 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 716e0c4dab584788bbb7631b4ca934bf
W20250625 07:52:41.071899 985 leader_election.cc:343] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 716e0c4dab584788bbb7631b4ca934bf
I20250625 07:52:41.072014 985 leader_election.cc:304] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.078444 2661 raft_consensus.cc:2747] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.078931 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9a0629f60a8e4eb999d459ff1d163f72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.079521 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a0629f60a8e4eb999d459ff1d163f72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.083460 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_04ba3f20d1184e0c9f9deb21ea849289" } new_table_name: "default.Table_be2966fdef3847bf8ac43fb4b6d15dbf" modify_external_catalogs: true
I20250625 07:52:41.083622 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bc8f47541c54c4287ebb7b892daa1be with delete_type TABLET_DATA_DELETED (replaced by tablet bb03e70e9aa740f18d8cd0543f730f83) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.087828 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f9f9fccc34685b73847c9d590ceb6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.089030 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b83f4c8544946d9adb3346df200b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:41.089429 850 leader_election.cc:343] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.089596 850 leader_election.cc:304] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.089843 2723 raft_consensus.cc:2747] T 813f9f9fccc34685b73847c9d590ceb6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.096858 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1352) took 1843 ms. Trace:
I20250625 07:52:41.099342 1161 rpcz_store.cc:276] 0625 07:52:39.253454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.253507 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:41.001786 (+1748279us) ts_tablet_manager.cc:1145] Deleting tablet cf175e17865e40ceb26aa955b91c2c13
0625 07:52:41.001797 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.018643 (+ 16846us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.021883 (+ 3240us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.096831 (+ 74948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748097}
I20250625 07:52:41.100070 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet cf175e17865e40ceb26aa955b91c2c13 (table default.Table_b674a39d7f9f44708b1369a5affecc82 [id=53aa2094bb3942c284fb15efcbc46bf9]) successfully deleted
I20250625 07:52:41.100576 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1353) took 1801 ms. Trace:
I20250625 07:52:41.100737 1161 rpcz_store.cc:276] 0625 07:52:39.298749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.298797 (+ 48us) service_pool.cc:225] Handling call
0625 07:52:41.100432 (+1801635us) ts_tablet_manager.cc:1145] Deleting tablet 03d999941e2a4265a64df7ec106a89a7
0625 07:52:41.100446 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.100558 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1801464}
W20250625 07:52:41.101248 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 03d999941e2a4265a64df7ec106a89a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d999941e2a4265a64df7ec106a89a7 already in progress: creating tablet
I20250625 07:52:41.101595 1161 tablet_replica.cc:331] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.101825 1161 raft_consensus.cc:2241] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.102037 1161 raft_consensus.cc:2270] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.102669 1161 ts_tablet_manager.cc:1905] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.111268 2661 raft_consensus.cc:491] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.111419 2661 raft_consensus.cc:513] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.112051 2661 leader_election.cc:290] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.112993 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:41.113132 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2db37ece7a964f47911fe4fc7434cd07" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.113773 986 leader_election.cc:304] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.114655 2661 raft_consensus.cc:2747] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.123867 1161 ts_tablet_manager.cc:1918] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.124081 1161 log.cc:1199] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/fdad7dc1b27e4d87a747642a6d47230c
I20250625 07:52:41.124480 1161 ts_tablet_manager.cc:1939] T fdad7dc1b27e4d87a747642a6d47230c P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.124908 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1354) took 1810 ms. Trace:
I20250625 07:52:41.125056 1161 rpcz_store.cc:276] 0625 07:52:39.314530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.314598 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:41.101547 (+1786949us) ts_tablet_manager.cc:1145] Deleting tablet fdad7dc1b27e4d87a747642a6d47230c
0625 07:52:41.101557 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.108676 (+ 7119us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.123761 (+ 15085us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.124888 (+ 1127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786697}
I20250625 07:52:41.125667 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet fdad7dc1b27e4d87a747642a6d47230c (table default.Table_ba10e2f968a543d4a5173431533d7e72 [id=6da9e5a92547450b83830b27727c69ef]) successfully deleted
I20250625 07:52:41.126133 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1355) took 1785 ms. Trace:
I20250625 07:52:41.126263 1161 rpcz_store.cc:276] 0625 07:52:39.340246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.340300 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:41.126004 (+1785704us) ts_tablet_manager.cc:1145] Deleting tablet 0fe117874dda4408934ce4e1687c10d6
0625 07:52:41.126016 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.126117 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1785486}
W20250625 07:52:41.126734 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0fe117874dda4408934ce4e1687c10d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe117874dda4408934ce4e1687c10d6 already in progress: creating tablet
I20250625 07:52:41.127091 1161 tablet_replica.cc:331] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.127314 1161 raft_consensus.cc:2241] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.127561 1161 raft_consensus.cc:2270] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.127735 1160 ts_tablet_manager.cc:1918] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.128611 1161 ts_tablet_manager.cc:1905] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.134773 1161 ts_tablet_manager.cc:1918] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:41.134928 1161 log.cc:1199] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8fdc3220d53142faae41fd1f1500de56
I20250625 07:52:41.135351 1161 ts_tablet_manager.cc:1939] T 8fdc3220d53142faae41fd1f1500de56 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.135677 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1356) took 1697 ms. Trace:
I20250625 07:52:41.135756 1161 rpcz_store.cc:276] 0625 07:52:39.437990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.438056 (+ 66us) service_pool.cc:225] Handling call
0625 07:52:41.127047 (+1688991us) ts_tablet_manager.cc:1145] Deleting tablet 8fdc3220d53142faae41fd1f1500de56
0625 07:52:41.127056 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.131724 (+ 4668us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.134680 (+ 2956us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.135659 (+ 979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1688745}
I20250625 07:52:41.136245 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]) successfully deleted
I20250625 07:52:41.147406 593 master_service.cc:948] ReplaceTablet: received request to replace tablet f2909c29db4c434c9db393c12ea48a1d from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:41.150781 1160 log.cc:1199] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/8f7240577a874a59b3268ff7f0c1beec
I20250625 07:52:41.151237 1160 ts_tablet_manager.cc:1939] T 8f7240577a874a59b3268ff7f0c1beec P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:41.152024 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1395) took 1527 ms. Trace:
I20250625 07:52:41.152192 1160 rpcz_store.cc:276] 0625 07:52:39.624300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.624341 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:40.975360 (+1351019us) ts_tablet_manager.cc:1145] Deleting tablet 8f7240577a874a59b3268ff7f0c1beec
0625 07:52:40.975370 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.124744 (+149374us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.127646 (+ 2902us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.152004 (+ 24358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350834}
I20250625 07:52:41.152825 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1397) took 1522 ms. Trace:
I20250625 07:52:41.152983 1160 rpcz_store.cc:276] 0625 07:52:39.629932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.629993 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:41.152658 (+1522665us) ts_tablet_manager.cc:1145] Deleting tablet b9eb3bcbece34f14a05dc7d95ca7ece2
0625 07:52:41.152668 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.152809 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522454}
I20250625 07:52:41.141510 1163 tablet_bootstrap.cc:654] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.159626 1160 tablet_replica.cc:331] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:41.159893 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 8f7240577a874a59b3268ff7f0c1beec (table default.Table_98c9cffe906a4a99bedb93374dab75e4 [id=f453bd82f85947dcb5d1e309ca1b902f]) successfully deleted
I20250625 07:52:41.159983 1160 raft_consensus.cc:2241] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:41.160179 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9eb3bcbece34f14a05dc7d95ca7ece2 already in progress: creating tablet
I20250625 07:52:41.160396 1160 raft_consensus.cc:2270] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.161172 1160 ts_tablet_manager.cc:1905] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.170044 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e24931f1a5c4745b7a6a7d204738c4e" } modify_external_catalogs: true
I20250625 07:52:41.170238 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e24931f1a5c4745b7a6a7d204738c4e" } modify_external_catalogs: true
W20250625 07:52:41.171593 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec2459d73e594a0c9622acc14bf1c5b5 consensus state has no leader
I20250625 07:52:41.171977 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a648db116d7d43969d0cf3d5c2501450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.172573 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a648db116d7d43969d0cf3d5c2501450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.174014 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_8ba515bb0f4d404da06248724a562452"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.174515 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ba515bb0f4d404da06248724a562452 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.175336 740 tablet_service.cc:1468] Processing CreateTablet for tablet f128431f6d0d4b2d8816ae169f9613d1 (DEFAULT_TABLE table=default.Table_6e61feb39d6145c0936cfa2316ddfdab [id=1b92149f05f74f99b64e1e608d5ea963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.175737 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f128431f6d0d4b2d8816ae169f9613d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.175742 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac8968da295c4b7fb6d83b84935d27d2 consensus state has no leader
I20250625 07:52:41.176041 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b1925e38730243d2bacf4c3b8ae37393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.176496 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1925e38730243d2bacf4c3b8ae37393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:41.177639 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edc15abe1a4b46b693167c8fda7f9ce1 consensus state has no leader
I20250625 07:52:41.177940 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f16cca6477ee43538ba95d01134aa5a3" } new_table_name: "default.Table_8a96bad446734b5486ca5eabf3a1901a" modify_external_catalogs: true
I20250625 07:52:41.179338 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ba58cf7f3f5943dc8d14d2f29fdb1684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.179818 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba58cf7f3f5943dc8d14d2f29fdb1684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.179873 739 tablet_service.cc:1468] Processing CreateTablet for tablet ae1300de7b364698935bdebdc6727c4a (DEFAULT_TABLE table=default.Table_42d926918dbf4b398f69aa2b3504f1b7 [id=c9a31a556fbe44589ce25c502c9fd469]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.180246 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1300de7b364698935bdebdc6727c4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.180503 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1842b67d00e248c59e02c1603a8c5300 consensus state has no leader
I20250625 07:52:41.180716 591 master_service.cc:948] ReplaceTablet: received request to replace tablet dba8780a44864169ac9cd63ab2e89fe9 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:41.185253 741 tablet_service.cc:1468] Processing CreateTablet for tablet fb22dd02b6d44396a13710f6463f1b93 (DEFAULT_TABLE table=default.Table_ca9bb00cf9834f0797220f17c7ad930d [id=247137e23c204848ab7f3362db576401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.185673 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb22dd02b6d44396a13710f6463f1b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.193681 754 tablet_service.cc:1468] Processing CreateTablet for tablet 60d2375fda2442cb9c566a2e640f0934 (DEFAULT_TABLE table=default.Table_d25be557c62b486592bc6f4c1025354e [id=28fb7ed6a3f348fbbf214380e3add79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.194139 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d2375fda2442cb9c566a2e640f0934. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.199645 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a571983779854eb398b3b0b42c34f83a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.200203 744 tablet_service.cc:1468] Processing CreateTablet for tablet 21d11cdcccb54af1ab68b39f9f0474a0 (DEFAULT_TABLE table=default.Table_cfcdd655f66c4381b702e1d689f72291 [id=e31dd4d7a11f4318ad37825530b1f794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.200726 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d11cdcccb54af1ab68b39f9f0474a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.202272 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c8694f09dcf47ab9f5fc8854c21c331 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.202479 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a396bde4b5964697ad0090eab777ba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.211268 1162 tablet_bootstrap.cc:492] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.211429 1162 ts_tablet_manager.cc:1397] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.330s user 0.001s sys 0.001s
I20250625 07:52:41.212177 1162 raft_consensus.cc:357] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.212363 1162 raft_consensus.cc:383] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.212420 1162 raft_consensus.cc:738] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.212646 1162 consensus_queue.cc:260] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.213595 1162 ts_tablet_manager.cc:1428] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:41.213873 1162 tablet_bootstrap.cc:492] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.215910 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 670974e43d7e4886bfe5eb3cf36ab6ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.216090 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b83f4c8544946d9adb3346df200b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.222959 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e15c7ac8ff934a8ca40ac480c7c352ab with delete_type TABLET_DATA_DELETED (replaced by tablet 1021a582ade04f80ba8e582e5040a217) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.199645 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a642a958550a40a48453f47d01b062a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.223407 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d43d98c0c51b40c6bdb0c2705465038e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.224076 2723 raft_consensus.cc:491] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.224241 2723 raft_consensus.cc:513] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.224946 2723 leader_election.cc:290] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:41.225652 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65231ee1a59d49eb96ba352b74c3460c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.226035 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3488f79d09054f6dbfc6b54c1dee0d6c" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:41.226563 850 leader_election.cc:343] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.226542 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65231ee1a59d49eb96ba352b74c3460c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:41.227034 851 leader_election.cc:343] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.227128 851 leader_election.cc:304] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.227695 2723 raft_consensus.cc:2747] T 65231ee1a59d49eb96ba352b74c3460c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.228538 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 8bc8f47541c54c4287ebb7b892daa1be (table default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]) successfully deleted
I20250625 07:52:41.232123 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet f2909c29db4c434c9db393c12ea48a1d
I20250625 07:52:41.230656 748 tablet_service.cc:1468] Processing CreateTablet for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3 (DEFAULT_TABLE table=default.Table_68221d261cec4a288a02a90ac62d03de [id=fd5cdd850e6a47a48c0623d71352fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.232681 593 catalog_manager.cc:6652] ReplaceTablet: tablet f2909c29db4c434c9db393c12ea48a1d deleted and replaced by tablet 595cd3f42a624ed3970d7ba8052a6d60
I20250625 07:52:41.232975 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6bbb1a2ead4b6382b33d2e83ce11c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.234284 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f62f60f959ba405b8f9f91731541cee3" } modify_external_catalogs: true
I20250625 07:52:41.234519 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f62f60f959ba405b8f9f91731541cee3" } modify_external_catalogs: true
I20250625 07:52:41.237556 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet f2909c29db4c434c9db393c12ea48a1d with delete_type TABLET_DATA_DELETED (replaced by tablet 595cd3f42a624ed3970d7ba8052a6d60) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.245119 884 tablet_service.cc:1515] Processing DeleteTablet for tablet f2909c29db4c434c9db393c12ea48a1d with delete_type TABLET_DATA_DELETED (replaced by tablet 595cd3f42a624ed3970d7ba8052a6d60) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.252215 1048 raft_consensus.cc:3058] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.253105 1158 tablet_bootstrap.cc:654] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.254204 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fd6c3a490883477dbafcb74d39be7345 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.255676 2661 raft_consensus.cc:491] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.255755 1046 raft_consensus.cc:3058] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.255817 2661 raft_consensus.cc:513] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.256428 2661 leader_election.cc:290] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.257617 985 leader_election.cc:343] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e4b87309b76f4f0e9dde120d9f755c44
I20250625 07:52:41.258888 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c8694f09dcf47ab9f5fc8854c21c331" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.259368 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4b87309b76f4f0e9dde120d9f755c44" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.259999 986 leader_election.cc:304] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.260834 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9f1d251171314118b66e3a921a4a50c5 (table default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]) successfully deleted
I20250625 07:52:41.261309 2661 raft_consensus.cc:2747] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.263424 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f31104720614b028e373ae1500c31cc" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.270701 2407 ts_tablet_manager.cc:1918] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:41.274771 1160 ts_tablet_manager.cc:1918] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.275285 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 70094e6c7e644f62bf5ef04dcfb4c380 (table default.Table_baabb7e0fd1e496bbe725e5e00574146 [id=c55921e26548445d9872a9e958f36056]) successfully deleted
W20250625 07:52:41.275566 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 53bf5c84b12d4bb399c16eb6e77a28ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bf5c84b12d4bb399c16eb6e77a28ef already in progress: creating tablet
I20250625 07:52:41.277510 1158 tablet_bootstrap.cc:492] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:41.277680 1158 ts_tablet_manager.cc:1397] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.235s user 0.003s sys 0.000s
I20250625 07:52:41.278582 1158 raft_consensus.cc:357] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.278801 1158 raft_consensus.cc:383] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.278849 1158 raft_consensus.cc:738] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.279084 1158 consensus_queue.cc:260] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.279668 1158 ts_tablet_manager.cc:1428] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:41.279958 1158 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:41.283460 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:41.291983 584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6 -> Peer 789c06e7de2842a1b40e97e0778ab6c9 (127.0.104.61:43317): Couldn't send request to peer 789c06e7de2842a1b40e97e0778ab6c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:43317: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:41.296463 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c858e286fbba450e81c6cc55f0f2b613" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.299046 2407 log.cc:1199] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/32a9f4cf6838411fab06fa6d35ae9492
I20250625 07:52:41.303637 2407 ts_tablet_manager.cc:1939] T 32a9f4cf6838411fab06fa6d35ae9492 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.305015 1047 raft_consensus.cc:3058] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.307340 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e181a3f1ae2c49349718d842c166b1e6" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.290900 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:41.307783 1160 log.cc:1199] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/5b04a575e0984f879f0179c8baa3f3fa
I20250625 07:52:41.307843 907 raft_consensus.cc:2466] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:41.311170 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet b3a84f7d816148788725216f713b3980
I20250625 07:52:41.311910 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b16a97fd65d14358a0b7bc51c14f5f61" } modify_external_catalogs: true
I20250625 07:52:41.312036 1160 ts_tablet_manager.cc:1939] T 5b04a575e0984f879f0179c8baa3f3fa P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:41.312075 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b16a97fd65d14358a0b7bc51c14f5f61" } modify_external_catalogs: true
I20250625 07:52:41.312870 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1396) took 1687 ms. Trace:
I20250625 07:52:41.313982 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1415) took 2845 ms. Trace:
W20250625 07:52:41.314694 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 909989e2e35a4a20a31d286b53c9ce8e consensus state has no leader
I20250625 07:52:41.315018 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_45dacbb132754fd98560d96145a5030f" } modify_external_catalogs: true
I20250625 07:52:41.315179 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_45dacbb132754fd98560d96145a5030f" } modify_external_catalogs: true
I20250625 07:52:41.315109 2407 rpcz_store.cc:276] 0625 07:52:38.468820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.468870 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:40.883788 (+2414918us) ts_tablet_manager.cc:1145] Deleting tablet 32a9f4cf6838411fab06fa6d35ae9492
0625 07:52:40.883796 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.171434 (+287638us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.270558 (+ 99124us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.313959 (+ 43401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2414772}
I20250625 07:52:41.316646 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1416) took 2847 ms. Trace:
I20250625 07:52:41.316749 2407 rpcz_store.cc:276] 0625 07:52:38.469185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.469234 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:41.316281 (+2847047us) ts_tablet_manager.cc:1145] Deleting tablet 7db12504a1a54add920561dae27d2d2b
0625 07:52:41.316294 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.316625 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846877}
I20250625 07:52:41.317423 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet dba8780a44864169ac9cd63ab2e89fe9
I20250625 07:52:41.317875 591 catalog_manager.cc:6652] ReplaceTablet: tablet dba8780a44864169ac9cd63ab2e89fe9 deleted and replaced by tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc
I20250625 07:52:41.318188 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_26f258ae9fd74416bba2b873db0695aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.318554 1158 tablet_bootstrap.cc:654] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:41.318699 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26f258ae9fd74416bba2b873db0695aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.319649 2407 tablet_replica.cc:331] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.319847 2407 raft_consensus.cc:2241] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:41.320164 1160 rpcz_store.cc:276] 0625 07:52:39.625553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.630533 (+ 4980us) service_pool.cc:225] Handling call
0625 07:52:41.159558 (+1529025us) ts_tablet_manager.cc:1145] Deleting tablet 5b04a575e0984f879f0179c8baa3f3fa
0625 07:52:41.159573 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.221671 (+ 62098us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.274623 (+ 52952us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.312834 (+ 38211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528774}
I20250625 07:52:41.321969 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f2c190af167a46ca8ba93fe63cc77209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.322376 2721 raft_consensus.cc:491] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:41.322521 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2c190af167a46ca8ba93fe63cc77209 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.322490 2721 raft_consensus.cc:513] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.323101 2721 leader_election.cc:290] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.323614 1160 tablet_replica.cc:331] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:41.324577 1160 raft_consensus.cc:2241] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:41.328516 713 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:41.328790 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b3a84f7d816148788725216f713b3980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.329110 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.329396 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 5b04a575e0984f879f0179c8baa3f3fa (table default.Table_bbe3a0ab3ea64ad1ba4b2ec8bb99fff1 [id=210e808ee8ce412888a041fa36c91cdb]) successfully deleted
I20250625 07:52:41.329710 884 tablet_service.cc:1515] Processing DeleteTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.329133 891 tablet_service.cc:1515] Processing DeleteTablet for tablet b3a84f7d816148788725216f713b3980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.338937 2745 tablet.cc:1722] T 2db37ece7a964f47911fe4fc7434cd07 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:41.340557 1160 raft_consensus.cc:2270] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.341749 1163 tablet_bootstrap.cc:492] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.342018 1163 ts_tablet_manager.cc:1397] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.332s user 0.013s sys 0.001s
I20250625 07:52:41.343006 1163 raft_consensus.cc:357] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.343304 1163 raft_consensus.cc:383] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.343420 1163 raft_consensus.cc:738] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.343791 1163 consensus_queue.cc:260] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.344609 1163 ts_tablet_manager.cc:1428] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:41.344955 1163 tablet_bootstrap.cc:492] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.346536 1163 tablet_bootstrap.cc:654] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.352984 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7744264852c74b1383d4855dcc2f149e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.353623 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7744264852c74b1383d4855dcc2f149e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.355293 2741 tablet.cc:1722] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:41.356599 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 96627us
I20250625 07:52:41.361414 597 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 2db37ece7a964f47911fe4fc7434cd07 (table default.Table_af477e9a0d964469882fdef9caf4ca71 [id=5dc8cc6762d04f4b924a1e2010186b19]). Expected version 3 got 0
I20250625 07:52:41.361124 1162 tablet_bootstrap.cc:654] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.361297 2407 pending_rounds.cc:70] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Trying to abort 1 pending ops.
I20250625 07:52:41.366159 2407 pending_rounds.cc:77] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171432271586967552 op_type: NO_OP noop_request { }
I20250625 07:52:41.366307 2407 raft_consensus.cc:2887] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250625 07:52:41.366405 2407 raft_consensus.cc:2270] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.367007 2407 ts_tablet_manager.cc:1905] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.374850 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d8ffd5ca03a642c699002e8361098a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.375620 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8ffd5ca03a642c699002e8361098a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.376878 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aa37d818be3b4a62ba68b85e674b01ab" } new_table_name: "default.Table_36a8b1fe2dd8489d90db18de6701477b" modify_external_catalogs: true
I20250625 07:52:41.379220 1160 ts_tablet_manager.cc:1905] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.380956 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1357) took 1902 ms. Trace:
I20250625 07:52:41.381059 1161 rpcz_store.cc:276] 0625 07:52:39.479602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.479651 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:41.380807 (+1901156us) ts_tablet_manager.cc:1145] Deleting tablet 35f819ef1fa74982a50d6714de305535
0625 07:52:41.380819 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.380931 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900972}
I20250625 07:52:41.381371 1161 tablet_replica.cc:331] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.381551 1161 raft_consensus.cc:2241] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.381721 1161 raft_consensus.cc:2270] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:41.381695 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 35f819ef1fa74982a50d6714de305535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f819ef1fa74982a50d6714de305535 already in progress: creating tablet
I20250625 07:52:41.382275 1161 ts_tablet_manager.cc:1905] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.384341 1158 tablet_bootstrap.cc:492] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:41.384476 1158 ts_tablet_manager.cc:1397] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.105s user 0.000s sys 0.003s
I20250625 07:52:41.385282 1158 raft_consensus.cc:357] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.385442 1158 raft_consensus.cc:383] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.385425 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e609e7267a66453dab2b666d69aebd02
I20250625 07:52:41.385490 1158 raft_consensus.cc:738] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.386116 589 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f20b6b4d24dd4bd388f75fc7dda4cd91
W20250625 07:52:41.386180 713 leader_election.cc:343] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3488f79d09054f6dbfc6b54c1dee0d6c
W20250625 07:52:41.386404 713 consensus_peers.cc:487] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 -> Peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Couldn't send request to peer 429026e8d98d46339db143f84897a092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4b87309b76f4f0e9dde120d9f755c44. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:41.388048 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet e609e7267a66453dab2b666d69aebd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.390214 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2166d8d523c34441943a1d577b745b4d" } new_table_name: "default.Table_8ffad117cd8945f2beee0e36d4809f1c" modify_external_catalogs: true
W20250625 07:52:41.391088 713 leader_election.cc:343] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 1c8694f09dcf47ab9f5fc8854c21c331
I20250625 07:52:41.391686 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_e8d8fca59ae7412bb768e66c8ab6e77f" } new_table_name: "default.Table_5bd347b575ac434b8316ddfd879697c3" modify_external_catalogs: true
I20250625 07:52:41.395296 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f90edcd94960433cb8e76ccabc3e4e59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.395784 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f90edcd94960433cb8e76ccabc3e4e59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.398226 891 tablet_service.cc:1515] Processing DeleteTablet for tablet e609e7267a66453dab2b666d69aebd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:41.399775 713 leader_election.cc:343] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9f31104720614b028e373ae1500c31cc
I20250625 07:52:41.399865 1162 tablet_bootstrap.cc:492] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.404721 1162 ts_tablet_manager.cc:1397] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.191s user 0.002s sys 0.001s
I20250625 07:52:41.405059 2749 tablet.cc:1722] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:41.405463 1162 raft_consensus.cc:357] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.405637 1162 raft_consensus.cc:383] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.405691 1162 raft_consensus.cc:738] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:41.405859 713 leader_election.cc:343] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c858e286fbba450e81c6cc55f0f2b613
I20250625 07:52:41.405885 1162 consensus_queue.cc:260] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.406500 1162 ts_tablet_manager.cc:1428] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:41.406523 2603 consensus_queue.cc:1035] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:41.406797 1162 tablet_bootstrap.cc:492] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.408507 1162 tablet_bootstrap.cc:654] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.416831 1163 tablet_bootstrap.cc:492] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.417097 1163 ts_tablet_manager.cc:1397] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.001s
I20250625 07:52:41.418104 1163 raft_consensus.cc:357] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.418350 1163 raft_consensus.cc:383] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.418460 1163 raft_consensus.cc:738] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.418524 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_4bd3fc265cec415ebcb3ecf6d4fa1f7a" } new_table_name: "default.Table_f8984bdccae14f52a39c0e273f526966" modify_external_catalogs: true
I20250625 07:52:41.418805 1163 consensus_queue.cc:260] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.420186 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_0e7144236c034538bb54d0c2d84c9b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.420671 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e7144236c034538bb54d0c2d84c9b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.420948 1163 ts_tablet_manager.cc:1428] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250625 07:52:41.421257 1163 tablet_bootstrap.cc:492] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.422818 1163 tablet_bootstrap.cc:654] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.423559 1158 consensus_queue.cc:260] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:41.423833 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec2459d73e594a0c9622acc14bf1c5b5 consensus state has no leader
I20250625 07:52:41.424116 1158 ts_tablet_manager.cc:1428] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250625 07:52:41.424105 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 487705c4b9ae4ab097771a69f03f05dc from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:41.424394 1158 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:41.426498 1158 tablet_bootstrap.cc:654] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.401003 2407 ts_tablet_manager.cc:1918] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:41.431932 2407 log.cc:1199] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:41.432498 2407 ts_tablet_manager.cc:1939] T 23af43b9ceeb4f2b812ee251ea8545ce P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.437637 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1417) took 2968 ms. Trace:
I20250625 07:52:41.437902 2407 rpcz_store.cc:276] 0625 07:52:38.469524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.469569 (+ 45us) service_pool.cc:225] Handling call
0625 07:52:41.319597 (+2850028us) ts_tablet_manager.cc:1145] Deleting tablet 23af43b9ceeb4f2b812ee251ea8545ce
0625 07:52:41.319608 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.397433 (+ 77825us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.400905 (+ 3472us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.437604 (+ 36699us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41115,"tablet-delete.queue_time_us":2849861}
I20250625 07:52:41.438745 1161 ts_tablet_manager.cc:1918] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.439056 2407 tablet_replica.cc:331] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.439316 2407 raft_consensus.cc:2241] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.439493 1161 log.cc:1199] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0aeb7077591f466eb77e3096f6b1d74b
I20250625 07:52:41.439697 2407 raft_consensus.cc:2270] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.440137 1161 ts_tablet_manager.cc:1939] T 0aeb7077591f466eb77e3096f6b1d74b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.440855 2407 ts_tablet_manager.cc:1905] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:41.441089 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1488eddcbbc84562a2bcb73099161de8 consensus state has no leader
I20250625 07:52:41.441282 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1358) took 1914 ms. Trace:
I20250625 07:52:41.441359 1161 rpcz_store.cc:276] 0625 07:52:39.526767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.527115 (+ 348us) service_pool.cc:225] Handling call
0625 07:52:41.381324 (+1854209us) ts_tablet_manager.cc:1145] Deleting tablet 0aeb7077591f466eb77e3096f6b1d74b
0625 07:52:41.381334 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.405930 (+ 24596us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.438613 (+ 32683us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.441265 (+ 2652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853920}
I20250625 07:52:41.443333 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0aeb7077591f466eb77e3096f6b1d74b (table default.Table_1225d003be2b4d048266736844a78a01 [id=357521ce57d54dafae9cfd2efbe731c2]) successfully deleted
W20250625 07:52:41.431934 713 leader_election.cc:343] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e181a3f1ae2c49349718d842c166b1e6
I20250625 07:52:41.444900 713 leader_election.cc:304] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:41.445318 2721 raft_consensus.cc:2802] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:41.445415 2721 raft_consensus.cc:491] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:41.445502 2721 raft_consensus.cc:3058] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.448843 2721 raft_consensus.cc:513] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.449450 2721 leader_election.cc:290] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.454326 1162 tablet_bootstrap.cc:492] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.454556 1162 ts_tablet_manager.cc:1397] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.006s
I20250625 07:52:41.455365 1162 raft_consensus.cc:357] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.455616 1162 raft_consensus.cc:383] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.455705 1162 raft_consensus.cc:738] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.455922 1162 consensus_queue.cc:260] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:41.456646 713 consensus_peers.cc:487] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:41.456828 1162 ts_tablet_manager.cc:1428] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:41.457134 1161 tablet_replica.cc:331] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:41.456847 713 consensus_peers.cc:487] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:41.457348 1161 raft_consensus.cc:2241] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.457528 1161 raft_consensus.cc:2270] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:41.458092 713 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.458256 713 leader_election.cc:304] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:41.458487 713 raft_consensus.cc:2684] T 9f1d251171314118b66e3a921a4a50c5 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:41.458474 1161 ts_tablet_manager.cc:1905] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:41.459198 713 leader_election.cc:343] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.459492 713 leader_election.cc:304] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.459759 2721 raft_consensus.cc:2747] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.459419 1162 tablet_bootstrap.cc:492] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.461822 1162 tablet_bootstrap.cc:654] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:41.463011 713 leader_election.cc:343] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.464136 713 leader_election.cc:304] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.464411 2603 raft_consensus.cc:2747] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.464960 713 leader_election.cc:343] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.465139 713 leader_election.cc:304] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.465400 2603 raft_consensus.cc:2747] T 9f31104720614b028e373ae1500c31cc P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.465790 713 consensus_peers.cc:487] T 0bfa1c4d4b6e4530892118189a5ab516 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:41.465991 713 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:41.466287 713 leader_election.cc:343] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.466423 713 leader_election.cc:304] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.466655 2603 raft_consensus.cc:2747] T c858e286fbba450e81c6cc55f0f2b613 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.470520 713 leader_election.cc:343] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.471135 713 leader_election.cc:304] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.471911 2603 raft_consensus.cc:2747] T e181a3f1ae2c49349718d842c166b1e6 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.473877 713 consensus_peers.cc:487] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fdc3220d53142faae41fd1f1500de56. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:41.474694 1160 ts_tablet_manager.cc:1918] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.474826 1160 log.cc:1199] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/22411725aa5741c497328ce047c44d50
I20250625 07:52:41.475234 1160 ts_tablet_manager.cc:1939] T 22411725aa5741c497328ce047c44d50 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:41.487382 1163 tablet_bootstrap.cc:492] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.487546 1163 ts_tablet_manager.cc:1397] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.000s
I20250625 07:52:41.488291 1163 raft_consensus.cc:357] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.488471 1163 raft_consensus.cc:383] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.488523 1163 raft_consensus.cc:738] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.489919 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 32a9f4cf6838411fab06fa6d35ae9492 (table default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]) successfully deleted
W20250625 07:52:41.490483 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7db12504a1a54add920561dae27d2d2b because the tablet was not found. No further retry: Not found: Tablet not found: 7db12504a1a54add920561dae27d2d2b
I20250625 07:52:41.492281 594 master_service.cc:948] ReplaceTablet: received request to replace tablet e9363b569fb84cc0bf55e0c6e2dbc608 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:41.492305 2745 tablet.cc:1722] T 173a565916c1440493ece348599ddba3 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:41.493520 1158 tablet_bootstrap.cc:492] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:41.493669 1158 ts_tablet_manager.cc:1397] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20250625 07:52:41.494333 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c82b840442ec4cb4993636183812ed22" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.494372 2407 ts_tablet_manager.cc:1918] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.494500 2407 log.cc:1199] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:41.494467 1158 raft_consensus.cc:357] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.494606 1158 raft_consensus.cc:383] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.494657 1158 raft_consensus.cc:738] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.494894 1158 consensus_queue.cc:260] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.495419 1158 ts_tablet_manager.cc:1428] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:41.496433 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1400) took 1845 ms. Trace:
I20250625 07:52:41.496729 1160 rpcz_store.cc:276] 0625 07:52:39.651099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.651162 (+ 63us) service_pool.cc:225] Handling call
0625 07:52:41.323555 (+1672393us) ts_tablet_manager.cc:1145] Deleting tablet 22411725aa5741c497328ce047c44d50
0625 07:52:41.323567 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.402971 (+ 79404us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.474573 (+ 71602us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.496413 (+ 21840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672149}
I20250625 07:52:41.497228 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 22411725aa5741c497328ce047c44d50 (table default.Table_41047b79cb784bb19f458dbce4573ef5 [id=772213882cb546a8bc37c0b8217eaa23]) successfully deleted
I20250625 07:52:41.497571 1160 tablet_replica.cc:331] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:41.497812 1160 raft_consensus.cc:2241] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.498164 1160 raft_consensus.cc:2270] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.506827 1158 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:41.513794 1161 ts_tablet_manager.cc:1918] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.513932 1161 log.cc:1199] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/07a07af928ff4d3d894b074ef0cf44eb
I20250625 07:52:41.514212 1161 ts_tablet_manager.cc:1939] T 07a07af928ff4d3d894b074ef0cf44eb P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.514602 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1359) took 1987 ms. Trace:
I20250625 07:52:41.514688 1161 rpcz_store.cc:276] 0625 07:52:39.527899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.528362 (+ 463us) service_pool.cc:225] Handling call
0625 07:52:41.457064 (+1928702us) ts_tablet_manager.cc:1145] Deleting tablet 07a07af928ff4d3d894b074ef0cf44eb
0625 07:52:41.457077 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.481652 (+ 24575us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.513682 (+ 32030us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.514586 (+ 904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1928474}
I20250625 07:52:41.514991 1161 tablet_replica.cc:331] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.515183 1161 raft_consensus.cc:2241] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.515357 1161 raft_consensus.cc:2270] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.515874 1161 ts_tablet_manager.cc:1905] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.516956 2407 ts_tablet_manager.cc:1939] T 9a90bb52358f4c38a540ad025e9e0430 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.518137 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1418) took 3048 ms. Trace:
I20250625 07:52:41.519176 2407 rpcz_store.cc:276] 0625 07:52:38.469829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.510157 (+ 40328us) service_pool.cc:225] Handling call
0625 07:52:41.439000 (+2928843us) ts_tablet_manager.cc:1145] Deleting tablet 9a90bb52358f4c38a540ad025e9e0430
0625 07:52:41.439011 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.471905 (+ 32894us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.494254 (+ 22349us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.518113 (+ 23859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2928616}
I20250625 07:52:41.522061 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1419) took 3011 ms. Trace:
I20250625 07:52:41.522477 2407 rpcz_store.cc:276] 0625 07:52:38.510653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.510712 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:41.521944 (+3011232us) ts_tablet_manager.cc:1145] Deleting tablet c263b671be384cd097041c60306034de
0625 07:52:41.521957 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.522042 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011007}
I20250625 07:52:41.522926 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1420) took 3011 ms. Trace:
I20250625 07:52:41.523041 2407 rpcz_store.cc:276] 0625 07:52:38.511119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.511168 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:41.522817 (+3011649us) ts_tablet_manager.cc:1145] Deleting tablet 4709f33f040149c49bd273524c9fd4f0
0625 07:52:41.522825 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.522912 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011487}
I20250625 07:52:41.523420 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1421) took 3012 ms. Trace:
I20250625 07:52:41.523535 2407 rpcz_store.cc:276] 0625 07:52:38.511458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.511504 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:41.523333 (+3011829us) ts_tablet_manager.cc:1145] Deleting tablet 3f1f025775d04da09eda17af4ac4fe10
0625 07:52:41.523341 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.523407 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011680}
I20250625 07:52:41.523886 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1429) took 2974 ms. Trace:
I20250625 07:52:41.524008 2407 rpcz_store.cc:276] 0625 07:52:38.549824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.549874 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:41.523779 (+2973905us) ts_tablet_manager.cc:1145] Deleting tablet 7d06e9aae7eb408c99d97e52fc9f691c
0625 07:52:41.523787 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.523873 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973750}
I20250625 07:52:41.524286 1160 ts_tablet_manager.cc:1905] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.524359 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1427) took 2975 ms. Trace:
I20250625 07:52:41.527004 2407 rpcz_store.cc:276] 0625 07:52:38.549304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.552734 (+ 3430us) service_pool.cc:225] Handling call
0625 07:52:41.524274 (+2971540us) ts_tablet_manager.cc:1145] Deleting tablet e8fa349c737f4cebb1225b20e62e3dcb
0625 07:52:41.524283 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.524348 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2971401}
I20250625 07:52:41.500825 1163 consensus_queue.cc:260] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.527647 1163 ts_tablet_manager.cc:1428] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250625 07:52:41.527988 1158 tablet_bootstrap.cc:654] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.528304 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1439) took 2690 ms. Trace:
I20250625 07:52:41.532541 1163 tablet_bootstrap.cc:492] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.534053 1163 tablet_bootstrap.cc:654] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.534193 2407 rpcz_store.cc:276] 0625 07:52:38.838289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.838581 (+ 292us) service_pool.cc:225] Handling call
0625 07:52:41.528190 (+2689609us) ts_tablet_manager.cc:1145] Deleting tablet 8a13381cae324e9494a2497d0190a868
0625 07:52:41.528200 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.528289 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2689429}
I20250625 07:52:41.534422 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_06f992952f9f4dc6b998e2be5694d564" } modify_external_catalogs: true
I20250625 07:52:41.534627 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_06f992952f9f4dc6b998e2be5694d564" } modify_external_catalogs: true
I20250625 07:52:41.534691 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1441) took 2696 ms. Trace:
I20250625 07:52:41.534773 2407 rpcz_store.cc:276] 0625 07:52:38.838454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.839154 (+ 700us) service_pool.cc:225] Handling call
0625 07:52:41.534562 (+2695408us) ts_tablet_manager.cc:1145] Deleting tablet 3a91007503ad43cc8f6fae30e4206df8
0625 07:52:41.534573 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.534672 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2695214}
I20250625 07:52:41.535197 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1442) took 2696 ms. Trace:
I20250625 07:52:41.535326 2407 rpcz_store.cc:276] 0625 07:52:38.838527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:38.839373 (+ 846us) service_pool.cc:225] Handling call
0625 07:52:41.534962 (+2695589us) ts_tablet_manager.cc:1145] Deleting tablet b4d308b0b16a4ab6a3a3589087c94136
0625 07:52:41.534971 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.535181 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2695480}
I20250625 07:52:41.535686 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1443) took 2395 ms. Trace:
I20250625 07:52:41.535827 2407 rpcz_store.cc:276] 0625 07:52:39.140428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.140737 (+ 309us) service_pool.cc:225] Handling call
0625 07:52:41.535592 (+2394855us) ts_tablet_manager.cc:1145] Deleting tablet 1536933d04014446ac8be0a896f0e6f3
0625 07:52:41.535601 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.535670 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2394677}
I20250625 07:52:41.535940 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_06a0b466c0dc496badbf8a1b18535c72" } new_table_name: "default.Table_3e7b17a7d5fc49d59f569ce163401177" modify_external_catalogs: true
I20250625 07:52:41.537274 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_323c4483e39f47e59c4b72b78dfdfc03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.537755 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_323c4483e39f47e59c4b72b78dfdfc03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.538929 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dcc596eca41947289921078947e54512"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.539381 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcc596eca41947289921078947e54512 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.540987 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1444) took 2400 ms. Trace:
I20250625 07:52:41.541064 2407 rpcz_store.cc:276] 0625 07:52:39.140512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.140949 (+ 437us) service_pool.cc:225] Handling call
0625 07:52:41.540854 (+2399905us) ts_tablet_manager.cc:1145] Deleting tablet ec1e58fe6d72451682bd47d7238560b6
0625 07:52:41.540865 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.540970 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399760}
I20250625 07:52:41.541389 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1445) took 2400 ms. Trace:
I20250625 07:52:41.541460 2407 rpcz_store.cc:276] 0625 07:52:39.140583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.141117 (+ 534us) service_pool.cc:225] Handling call
0625 07:52:41.541293 (+2400176us) ts_tablet_manager.cc:1145] Deleting tablet dbdef039d3f842e5957c5bab85bedd42
0625 07:52:41.541302 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.541374 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2400041}
I20250625 07:52:41.541707 2407 tablet_replica.cc:331] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.541903 2407 raft_consensus.cc:2241] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.542053 2407 raft_consensus.cc:2270] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.542616 2407 ts_tablet_manager.cc:1905] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.545498 2718 raft_consensus.cc:491] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.545639 2718 raft_consensus.cc:513] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.546227 2718 leader_election.cc:290] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.519086 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 07a07af928ff4d3d894b074ef0cf44eb (table default.Table_eb85d0d01edc400eabcdd8a8175dfbf9 [id=68ab97279148448991b34aeef51de39a]) successfully deleted
I20250625 07:52:41.549191 596 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 173a565916c1440493ece348599ddba3 (table default.Table_cb77d581902c4391ac8bcdca79163105 [id=b1ec5aff86df4e9381d955d406326bde]). Expected version 1 got 0
I20250625 07:52:41.551098 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_66c399c228be41b2888b89548b2d5738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.555368 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet c3d8b6b5c06248879e6052f6ae7fe46e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:41.558971 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66c399c228be41b2888b89548b2d5738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.561789 1163 tablet_bootstrap.cc:492] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.561934 1163 ts_tablet_manager.cc:1397] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20250625 07:52:41.562633 1163 raft_consensus.cc:357] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.562795 1163 raft_consensus.cc:383] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.562851 1163 raft_consensus.cc:738] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.563043 1163 consensus_queue.cc:260] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.565057 1163 ts_tablet_manager.cc:1428] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:41.567010 1163 tablet_bootstrap.cc:492] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.568451 1163 tablet_bootstrap.cc:654] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.572999 1162 tablet_bootstrap.cc:492] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.573159 1162 ts_tablet_manager.cc:1397] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.000s
W20250625 07:52:41.553791 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3522ef3ed0714b60944a851bad7375fc consensus state has no leader
I20250625 07:52:41.562036 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_89bf4acb52434e51a773e4ff8b98b203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.575297 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89bf4acb52434e51a773e4ff8b98b203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.576109 2407 ts_tablet_manager.cc:1918] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.576331 2407 log.cc:1199] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3488f79d09054f6dbfc6b54c1dee0d6c
I20250625 07:52:41.576611 2407 ts_tablet_manager.cc:1939] T 3488f79d09054f6dbfc6b54c1dee0d6c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.577183 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 5179a144a9a0445f933ca02e79bbc122 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:19 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.577288 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet a41ffd566593499299aba5513ccb2f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:29 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.577339 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1447) took 2436 ms. Trace:
I20250625 07:52:41.577406 2407 rpcz_store.cc:276] 0625 07:52:39.140743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.141579 (+ 836us) service_pool.cc:225] Handling call
0625 07:52:41.541659 (+2400080us) ts_tablet_manager.cc:1145] Deleting tablet 3488f79d09054f6dbfc6b54c1dee0d6c
0625 07:52:41.541668 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.550181 (+ 8513us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.575986 (+ 25805us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.577319 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399938}
I20250625 07:52:41.577755 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1448) took 2436 ms. Trace:
I20250625 07:52:41.578699 1163 tablet_bootstrap.cc:492] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.578848 1163 ts_tablet_manager.cc:1397] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250625 07:52:41.579478 1163 raft_consensus.cc:357] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.579634 1163 raft_consensus.cc:383] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.579685 1163 raft_consensus.cc:738] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.579872 1163 consensus_queue.cc:260] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.580314 1163 ts_tablet_manager.cc:1428] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:41.579849 2407 rpcz_store.cc:276] 0625 07:52:39.140823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.141752 (+ 929us) service_pool.cc:225] Handling call
0625 07:52:41.577630 (+2435878us) ts_tablet_manager.cc:1145] Deleting tablet 60c01f0a75674c469b41fac638799afa
0625 07:52:41.577639 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.577740 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2435724}
I20250625 07:52:41.580580 1163 tablet_bootstrap.cc:492] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.581866 1163 tablet_bootstrap.cc:654] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.582078 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1449) took 2242 ms. Trace:
I20250625 07:52:41.582228 2407 rpcz_store.cc:276] 0625 07:52:39.339505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.340016 (+ 511us) service_pool.cc:225] Handling call
0625 07:52:41.580584 (+2240568us) ts_tablet_manager.cc:1145] Deleting tablet 03d999941e2a4265a64df7ec106a89a7
0625 07:52:41.580595 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.582057 (+ 1462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2222937}
I20250625 07:52:41.583998 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:41.584525 2723 raft_consensus.cc:491] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.584645 2723 raft_consensus.cc:513] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.585248 2723 leader_election.cc:290] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:41.586277 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f771628379842989b9b2099de65e2a0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:41.589609 851 leader_election.cc:343] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.589988 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2fa277660d76420a834714f10f39c3af" } modify_external_catalogs: true
I20250625 07:52:41.590224 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2fa277660d76420a834714f10f39c3af" } modify_external_catalogs: true
I20250625 07:52:41.592983 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_12ba62aefbe74bafa70d60756d1e0b0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.593508 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12ba62aefbe74bafa70d60756d1e0b0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.594183 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1450) took 2249 ms. Trace:
I20250625 07:52:41.594292 2407 rpcz_store.cc:276] 0625 07:52:39.339609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.412761 (+ 73152us) service_pool.cc:225] Handling call
0625 07:52:41.589049 (+2176288us) ts_tablet_manager.cc:1145] Deleting tablet 0fe117874dda4408934ce4e1687c10d6
0625 07:52:41.589063 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.589263 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176038}
I20250625 07:52:41.597977 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f21fd181ae64529a96b8dd7b83c8cad" } modify_external_catalogs: true
I20250625 07:52:41.598268 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2f21fd181ae64529a96b8dd7b83c8cad" } modify_external_catalogs: true
I20250625 07:52:41.607525 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:41.607933 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f771628379842989b9b2099de65e2a0" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:41.609232 850 leader_election.cc:343] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.609375 850 leader_election.cc:304] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.609776 2723 raft_consensus.cc:2747] T 1f771628379842989b9b2099de65e2a0 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.609741 1162 raft_consensus.cc:357] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.609927 1162 raft_consensus.cc:383] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.609987 1162 raft_consensus.cc:738] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.610164 1162 consensus_queue.cc:260] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.610653 1162 ts_tablet_manager.cc:1428] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20250625 07:52:41.610963 1162 tablet_bootstrap.cc:492] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.611469 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1f771628379842989b9b2099de65e2a0 (table default.Table_2abbb2dbe82c49db85360a028645c2bd [id=63a78517d1cb4d6790a9422cb7b822aa]). Expected version 1 got 0
I20250625 07:52:41.594610 2407 tablet_replica.cc:331] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.612924 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 9eaab8f86bdc45cab8645cd5a6e3489c (table default.Table_5090762d485f41399e8ac22e3317f483 [id=54e5a28725fb48609e29a7a01c76a3ed]). Expected version 1 got 0
I20250625 07:52:41.613224 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:41.614938 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:41.614094 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 23af43b9ceeb4f2b812ee251ea8545ce (table default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]) successfully deleted
I20250625 07:52:41.614027 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:41.617244 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 57c3d0810cf44a33b2acd50707931d26 (table default.Table_e7eadb1ed2af45bf8ebc908ee6e8b073 [id=e54da0da99f744839ff8cf175e23efda]). Expected version 2 got 0
I20250625 07:52:41.617484 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_3e4796a533d44263a8627ae547f7c8ba [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 3 got 0
I20250625 07:52:41.618659 591 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet d7ed91509ba94562a9207c7d7c0d3e2b (table default.Table_ca90d59a48d947778cd899359e52ed41 [id=1353496ce19c4b14bdcc41400f3e9844]). Expected version 1 got 0
I20250625 07:52:41.618914 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6f04229a1d8462c818d589f9f4eb712" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:41.619155 907 raft_consensus.cc:3058] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.622025 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1c150fb54c405c8dc61ac133d98b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:27 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.622948 907 raft_consensus.cc:2466] T d6f04229a1d8462c818d589f9f4eb712 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:41.624092 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bdce2b3069f1428487769ca6ac55ff24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.625139 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdce2b3069f1428487769ca6ac55ff24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.626510 2745 tablet.cc:1722] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:41.629616 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 023b6db7592a4c62800a2d718c9ce339 (DEFAULT_TABLE table=default.Table_a47f7fe519514bb0a711481e766950b2 [id=f8a101c4976a488c94a4c1eeb317712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.630323 891 tablet_service.cc:1468] Processing CreateTablet for tablet 023b6db7592a4c62800a2d718c9ce339 (DEFAULT_TABLE table=default.Table_a47f7fe519514bb0a711481e766950b2 [id=f8a101c4976a488c94a4c1eeb317712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.631171 1015 tablet_service.cc:1468] Processing CreateTablet for tablet f325a69f85b947ffa434323dbada1eed (DEFAULT_TABLE table=default.Table_327aebe35f38451ba0f50824bef20577 [id=8b36ca34e5fb4874b88ad129611fc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.631405 884 tablet_service.cc:1468] Processing CreateTablet for tablet f325a69f85b947ffa434323dbada1eed (DEFAULT_TABLE table=default.Table_327aebe35f38451ba0f50824bef20577 [id=8b36ca34e5fb4874b88ad129611fc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.632944 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f325a69f85b947ffa434323dbada1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.633132 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b6db7592a4c62800a2d718c9ce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.649518 1163 tablet_bootstrap.cc:492] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.649842 1163 ts_tablet_manager.cc:1397] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.069s user 0.019s sys 0.032s
I20250625 07:52:41.650543 1163 raft_consensus.cc:357] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.650705 1163 raft_consensus.cc:383] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.650753 1163 raft_consensus.cc:738] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.650954 1163 consensus_queue.cc:260] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.651556 1163 ts_tablet_manager.cc:1428] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:41.651919 1163 tablet_bootstrap.cc:492] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.653549 1163 tablet_bootstrap.cc:654] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.653846 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 487705c4b9ae4ab097771a69f03f05dc
I20250625 07:52:41.657999 592 catalog_manager.cc:6652] ReplaceTablet: tablet 487705c4b9ae4ab097771a69f03f05dc deleted and replaced by tablet 7b112baddf9e4da3902734fc3fe31e9a
I20250625 07:52:41.658278 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 487705c4b9ae4ab097771a69f03f05dc with delete_type TABLET_DATA_DELETED (replaced by tablet 7b112baddf9e4da3902734fc3fe31e9a) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.660286 2755 tablet.cc:1722] T 2082e5e28c5c4d1fac7e40acf77f75e6 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250625 07:52:41.661425 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b6db7592a4c62800a2d718c9ce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.664978 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f325a69f85b947ffa434323dbada1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.656304 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 565d312783f24b9b995595f413ce4697
I20250625 07:52:41.656738 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet ac8968da295c4b7fb6d83b84935d27d2
I20250625 07:52:41.656874 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet acfea07d8441415e92d59f2f4e0b72a6
I20250625 07:52:41.657135 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4ee3563a2d994e6e88d2e0715a7b3816
I20250625 07:52:41.669919 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.670748 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.670764 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.677062 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_95a02ee5862445a89fcc2bc41403f418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.679780 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_da2cc3d0fc2a4550847f4606db8c777a" } new_table_name: "default.Table_b002c2ea742b48d9bc76f129905d3bee" modify_external_catalogs: true
I20250625 07:52:41.679987 884 tablet_service.cc:1515] Processing DeleteTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.681741 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7918799ea32a4b6bbd7c95a79962e5f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.681872 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:41.682238 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7918799ea32a4b6bbd7c95a79962e5f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.685438 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1b8a59feb1b443b2899089b11bdb6a2f" } new_table_name: "default.Table_f21f8e1c5213408aa76375a01f4ff249" modify_external_catalogs: true
I20250625 07:52:41.687172 1161 ts_tablet_manager.cc:1918] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.687203 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 565d312783f24b9b995595f413ce4697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.687255 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 565d312783f24b9b995595f413ce4697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.687312 1161 log.cc:1199] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0657163b9a0d4f7f99f57f7fd1f32462
I20250625 07:52:41.687657 1161 ts_tablet_manager.cc:1939] T 0657163b9a0d4f7f99f57f7fd1f32462 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.687637 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cfcdd655f66c4381b702e1d689f72291" } new_table_name: "default.Table_0f1a50893ade4b108b260a6a2b3b90b2" modify_external_catalogs: true
I20250625 07:52:41.687968 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1361) took 2157 ms. Trace:
I20250625 07:52:41.688169 884 tablet_service.cc:1515] Processing DeleteTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.688038 1161 rpcz_store.cc:276] 0625 07:52:39.530430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.530630 (+ 200us) service_pool.cc:225] Handling call
0625 07:52:41.514948 (+1984318us) ts_tablet_manager.cc:1145] Deleting tablet 0657163b9a0d4f7f99f57f7fd1f32462
0625 07:52:41.514957 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.537182 (+ 22225us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.687043 (+149861us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.687953 (+ 910us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984138}
I20250625 07:52:41.688416 1161 tablet_replica.cc:331] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.688513 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0657163b9a0d4f7f99f57f7fd1f32462 (table default.Table_7476f8055aa048e9b8c30b2269f4f2c9 [id=904f631d8b704232b9182be1fa4aff8b]) successfully deleted
I20250625 07:52:41.688601 1161 raft_consensus.cc:2241] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:41.689662 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95a02ee5862445a89fcc2bc41403f418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.690716 1160 ts_tablet_manager.cc:1918] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.690843 1160 log.cc:1199] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/d0a75906385744e5ba6938f451a2d7ec
I20250625 07:52:41.691139 1160 ts_tablet_manager.cc:1939] T d0a75906385744e5ba6938f451a2d7ec P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:41.691382 1161 raft_consensus.cc:2270] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.691838 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1413) took 1906 ms. Trace:
I20250625 07:52:41.691937 1160 rpcz_store.cc:276] 0625 07:52:39.785462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.787295 (+ 1833us) service_pool.cc:225] Handling call
0625 07:52:41.497510 (+1710215us) ts_tablet_manager.cc:1145] Deleting tablet d0a75906385744e5ba6938f451a2d7ec
0625 07:52:41.497520 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.640240 (+142720us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.690602 (+ 50362us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.691817 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768,"tablet-delete.queue_time_us":1709989}
I20250625 07:52:41.692428 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet d0a75906385744e5ba6938f451a2d7ec (table default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]) successfully deleted
I20250625 07:52:41.692850 1161 ts_tablet_manager.cc:1905] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.693881 2736 raft_consensus.cc:491] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.693884 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 086b8931d4c14af08518e088dc56ad20 (DEFAULT_TABLE table=default.Table_5976d462ca1440b6b16932b369bfa86a [id=b7682b02c75e479a842995eb055d00cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.694048 2736 raft_consensus.cc:513] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.694324 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b8931d4c14af08518e088dc56ad20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.694609 2736 leader_election.cc:290] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.696121 986 leader_election.cc:343] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:41.696324 2736 raft_consensus.cc:491] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.696420 2736 raft_consensus.cc:513] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.696951 2736 leader_election.cc:290] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.697844 985 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: da895c90d6ee40379fe8181c2dbd7a1e
W20250625 07:52:41.698371 986 leader_election.cc:343] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: da895c90d6ee40379fe8181c2dbd7a1e
I20250625 07:52:41.698460 986 leader_election.cc:304] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.698855 2736 raft_consensus.cc:2747] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.701740 985 leader_election.cc:343] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:41.701848 985 leader_election.cc:304] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.702090 2736 raft_consensus.cc:2747] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.702919 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 363fee07295c44139fd713b33c82623c
I20250625 07:52:41.703804 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cbe74ec8a2f443a4afe33052ace1ce64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.703913 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 363fee07295c44139fd713b33c82623c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:41.704355 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbe74ec8a2f443a4afe33052ace1ce64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.612668 1162 tablet_bootstrap.cc:654] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.612946 2407 raft_consensus.cc:2241] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:41.707491 884 tablet_service.cc:1468] Processing CreateTablet for tablet 086b8931d4c14af08518e088dc56ad20 (DEFAULT_TABLE table=default.Table_5976d462ca1440b6b16932b369bfa86a [id=b7682b02c75e479a842995eb055d00cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.707845 2407 raft_consensus.cc:2270] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.707890 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b8931d4c14af08518e088dc56ad20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.708397 2407 ts_tablet_manager.cc:1905] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.624224 748 tablet_service.cc:1515] Processing DeleteTablet for tablet f2909c29db4c434c9db393c12ea48a1d with delete_type TABLET_DATA_DELETED (replaced by tablet 595cd3f42a624ed3970d7ba8052a6d60) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.648217 744 tablet_service.cc:1515] Processing DeleteTablet for tablet b3a84f7d816148788725216f713b3980 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.648480 752 tablet_service.cc:1515] Processing DeleteTablet for tablet dba8780a44864169ac9cd63ab2e89fe9 with delete_type TABLET_DATA_DELETED (replaced by tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.677150 740 tablet_service.cc:1468] Processing CreateTablet for tablet 023b6db7592a4c62800a2d718c9ce339 (DEFAULT_TABLE table=default.Table_a47f7fe519514bb0a711481e766950b2 [id=f8a101c4976a488c94a4c1eeb317712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.712589 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023b6db7592a4c62800a2d718c9ce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.714114 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 487705c4b9ae4ab097771a69f03f05dc with delete_type TABLET_DATA_DELETED (replaced by tablet 7b112baddf9e4da3902734fc3fe31e9a) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.715178 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 565d312783f24b9b995595f413ce4697 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.687144 741 tablet_service.cc:1468] Processing CreateTablet for tablet f325a69f85b947ffa434323dbada1eed (DEFAULT_TABLE table=default.Table_327aebe35f38451ba0f50824bef20577 [id=8b36ca34e5fb4874b88ad129611fc730]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.715764 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f325a69f85b947ffa434323dbada1eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.715731 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0352225d645c408ba8c74273ee5f56e6 (table default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a]). Expected version 2 got 0
I20250625 07:52:41.715888 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 9a90bb52358f4c38a540ad025e9e0430 (table default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]) successfully deleted
I20250625 07:52:41.715962 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
W20250625 07:52:41.716231 584 catalog_manager.cc:4895] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet c263b671be384cd097041c60306034de because the tablet was not found. No further retry: Not found: Tablet not found: c263b671be384cd097041c60306034de
I20250625 07:52:41.716398 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d7ed91509ba94562a9207c7d7c0d3e2b (table default.Table_ca90d59a48d947778cd899359e52ed41 [id=1353496ce19c4b14bdcc41400f3e9844]). Expected version 1 got 0
W20250625 07:52:41.716532 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4709f33f040149c49bd273524c9fd4f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4709f33f040149c49bd273524c9fd4f0 already in progress: creating tablet
I20250625 07:52:41.716591 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]). Expected version 1 got 0
I20250625 07:52:41.718391 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:41.719291 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 15766b87fb1b4620970d5fab21465cc4 (table default.Table_5bd347b575ac434b8316ddfd879697c3 [id=d4be884fab4f4468a41fb5a7eff91ad0]). Expected version 1 got 0
I20250625 07:52:41.719473 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_3e4796a533d44263a8627ae547f7c8ba [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 3 got 0
I20250625 07:52:41.719837 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]). Expected version 2 got 0
I20250625 07:52:41.720194 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:41.687822 750 tablet_service.cc:1515] Processing DeleteTablet for tablet acfea07d8441415e92d59f2f4e0b72a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.687963 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee3563a2d994e6e88d2e0715a7b3816 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:40 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.649416 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e609e7267a66453dab2b666d69aebd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.687700 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 487705c4b9ae4ab097771a69f03f05dc with delete_type TABLET_DATA_DELETED (replaced by tablet 7b112baddf9e4da3902734fc3fe31e9a) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.724737 891 tablet_service.cc:1468] Processing CreateTablet for tablet 71edb6f95ed74e1c81afe85d08638002 (DEFAULT_TABLE table=default.Table_7e5a70063453457485e565593de59bec [id=03b14f087cd64a39b8c19c978516b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.724906 2736 raft_consensus.cc:491] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.725033 2736 raft_consensus.cc:513] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.725171 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71edb6f95ed74e1c81afe85d08638002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.725600 2736 leader_election.cc:290] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.725957 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5239065a44e24cf28772215a5fbab86e (DEFAULT_TABLE table=default.Table_1b8a59feb1b443b2899089b11bdb6a2f [id=9e3863c1c2024ecbb67db85aad410b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.726279 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5239065a44e24cf28772215a5fbab86e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.727000 986 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: a41ffd566593499299aba5513ccb2f30
W20250625 07:52:41.728547 985 leader_election.cc:343] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a41ffd566593499299aba5513ccb2f30
I20250625 07:52:41.728653 985 leader_election.cc:304] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.729415 2736 raft_consensus.cc:2747] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:41.730050 581 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f54fa729d3754d7eb7e0528a733d40b6 (table default.Table_a9040aa37b69422fbf38c7059883fd02 [id=9f2e39e91a134dcb8fa2c9a0bf97cf36], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f54fa729d3754d7eb7e0528a733d40b6 consensus state has no leader
I20250625 07:52:41.733054 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b782414a682246feb2811b4e3bf8b5a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.736043 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 670f3b7f0a904656af418d01ed6f58b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.738147 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5239065a44e24cf28772215a5fbab86e (DEFAULT_TABLE table=default.Table_1b8a59feb1b443b2899089b11bdb6a2f [id=9e3863c1c2024ecbb67db85aad410b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.738808 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5239065a44e24cf28772215a5fbab86e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.738759 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3f1f025775d04da09eda17af4ac4fe10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f1f025775d04da09eda17af4ac4fe10 already in progress: creating tablet
W20250625 07:52:41.739243 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7d06e9aae7eb408c99d97e52fc9f691c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d06e9aae7eb408c99d97e52fc9f691c already in progress: creating tablet
W20250625 07:52:41.739833 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet e8fa349c737f4cebb1225b20e62e3dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8fa349c737f4cebb1225b20e62e3dcb already in progress: creating tablet
W20250625 07:52:41.740648 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 8a13381cae324e9494a2497d0190a868 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a13381cae324e9494a2497d0190a868 already in progress: creating tablet
W20250625 07:52:41.741503 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 3a91007503ad43cc8f6fae30e4206df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a91007503ad43cc8f6fae30e4206df8 already in progress: creating tablet
W20250625 07:52:41.741883 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b4d308b0b16a4ab6a3a3589087c94136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4d308b0b16a4ab6a3a3589087c94136 already in progress: creating tablet
W20250625 07:52:41.742229 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 1536933d04014446ac8be0a896f0e6f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1536933d04014446ac8be0a896f0e6f3 already in progress: creating tablet
W20250625 07:52:41.742583 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ec1e58fe6d72451682bd47d7238560b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec1e58fe6d72451682bd47d7238560b6 already in progress: creating tablet
W20250625 07:52:41.742918 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet dbdef039d3f842e5957c5bab85bedd42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbdef039d3f842e5957c5bab85bedd42 already in progress: creating tablet
I20250625 07:52:41.743170 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bbe3a0ab3ea64ad1ba4b2ec8bb99fff1" } new_table_name: "default.Table_746708b6e5dd49baafaee63cb0e3ecef" modify_external_catalogs: true
I20250625 07:52:41.745126 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3488f79d09054f6dbfc6b54c1dee0d6c (table default.Table_c121aa9cbde14c6bb4a4bcb574484208 [id=b6a77883382d4a6f8ff92139c24ba3be]) successfully deleted
I20250625 07:52:41.745280 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dddbc1200acf4e7fa93ef85b398d5832" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:41.745611 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 60c01f0a75674c469b41fac638799afa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60c01f0a75674c469b41fac638799afa already in progress: creating tablet
W20250625 07:52:41.746037 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 03d999941e2a4265a64df7ec106a89a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d999941e2a4265a64df7ec106a89a7 already in progress: creating tablet
W20250625 07:52:41.746351 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 0fe117874dda4408934ce4e1687c10d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe117874dda4408934ce4e1687c10d6 already in progress: creating tablet
I20250625 07:52:41.747380 1048 raft_consensus.cc:3058] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.747731 1046 raft_consensus.cc:3058] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:41.749053 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 71edb6f95ed74e1c81afe85d08638002 (DEFAULT_TABLE table=default.Table_7e5a70063453457485e565593de59bec [id=03b14f087cd64a39b8c19c978516b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.749415 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71edb6f95ed74e1c81afe85d08638002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.753559 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 363fee07295c44139fd713b33c82623c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:41.754549 2407 ts_tablet_manager.cc:1918] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250625 07:52:41.754721 2407 log.cc:1199] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/8fdc3220d53142faae41fd1f1500de56
I20250625 07:52:41.755151 2407 ts_tablet_manager.cc:1939] T 8fdc3220d53142faae41fd1f1500de56 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.757586 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1452) took 2225 ms. Trace:
I20250625 07:52:41.760489 1158 tablet_bootstrap.cc:492] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:41.760641 1158 ts_tablet_manager.cc:1397] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.254s user 0.006s sys 0.002s
I20250625 07:52:41.761456 1158 raft_consensus.cc:357] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.761628 1158 raft_consensus.cc:383] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.761675 1158 raft_consensus.cc:738] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.761878 1158 consensus_queue.cc:260] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.762390 1158 ts_tablet_manager.cc:1428] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:41.762660 1158 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:41.763398 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e9363b569fb84cc0bf55e0c6e2dbc608
I20250625 07:52:41.763883 594 catalog_manager.cc:6652] ReplaceTablet: tablet e9363b569fb84cc0bf55e0c6e2dbc608 deleted and replaced by tablet ae25051c24024b5897d8115e032ca3b0
I20250625 07:52:41.765640 2407 rpcz_store.cc:276] 0625 07:52:39.532407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.536928 (+ 4521us) service_pool.cc:225] Handling call
0625 07:52:41.594561 (+2057633us) ts_tablet_manager.cc:1145] Deleting tablet 8fdc3220d53142faae41fd1f1500de56
0625 07:52:41.594573 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.730618 (+136045us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.754429 (+ 23811us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.757565 (+ 3136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2057418}
I20250625 07:52:41.766182 2407 tablet_replica.cc:331] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.766469 2407 raft_consensus.cc:2241] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.766754 594 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:41.767362 594 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 23908409fbbd48dbbe82be8182ba665f (table default.Table_705c8d46163b48cc9cf55cdf1aa9e887 [id=f3d401f6b64540f9a0594205585b22ee]). Expected version 1 got 0
I20250625 07:52:41.770602 2736 raft_consensus.cc:491] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.770745 2736 raft_consensus.cc:513] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.770787 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_106241d2c1a44caca30ba7a0a604059f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.771293 2736 leader_election.cc:290] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.771363 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_106241d2c1a44caca30ba7a0a604059f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.772533 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 with delete_type TABLET_DATA_DELETED (replaced by tablet ae25051c24024b5897d8115e032ca3b0) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:41.773628 986 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:41.775811 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 with delete_type TABLET_DATA_DELETED (replaced by tablet ae25051c24024b5897d8115e032ca3b0) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:41.776443 985 leader_election.cc:343] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:41.776561 985 leader_election.cc:304] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.776602 2407 raft_consensus.cc:2270] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.776942 2736 raft_consensus.cc:2747] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.794955 1160 tablet_replica.cc:331] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:41.795265 1160 raft_consensus.cc:2241] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.795490 1160 raft_consensus.cc:2270] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.796139 1160 ts_tablet_manager.cc:1905] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.801448 1161 ts_tablet_manager.cc:1918] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:41.806056 713 leader_election.cc:343] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.807423 1162 tablet_bootstrap.cc:492] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.807561 1162 ts_tablet_manager.cc:1397] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.197s user 0.000s sys 0.003s
I20250625 07:52:41.807657 2723 raft_consensus.cc:491] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.807788 2723 raft_consensus.cc:513] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:41.807942 713 leader_election.cc:343] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.808286 1162 raft_consensus.cc:357] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.808399 2723 leader_election.cc:290] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:41.808449 1162 raft_consensus.cc:383] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.808501 1162 raft_consensus.cc:738] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.808948 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b782414a682246feb2811b4e3bf8b5a2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.808713 1162 consensus_queue.cc:260] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:41.809500 851 leader_election.cc:343] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.812072 1162 ts_tablet_manager.cc:1428] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:41.812374 1162 tablet_bootstrap.cc:492] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.813824 1162 tablet_bootstrap.cc:654] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.814003 1161 log.cc:1199] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/0b7daf759e324c1db853fdbaab76c721
I20250625 07:52:41.809248 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b782414a682246feb2811b4e3bf8b5a2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:41.814473 1161 ts_tablet_manager.cc:1939] T 0b7daf759e324c1db853fdbaab76c721 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.814823 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1360) took 2286 ms. Trace:
W20250625 07:52:41.814891 850 leader_election.cc:343] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.815037 850 leader_election.cc:304] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.815428 2723 raft_consensus.cc:2747] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.817051 1161 rpcz_store.cc:276] 0625 07:52:39.528844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.530218 (+ 1374us) service_pool.cc:225] Handling call
0625 07:52:41.688372 (+2158154us) ts_tablet_manager.cc:1145] Deleting tablet 0b7daf759e324c1db853fdbaab76c721
0625 07:52:41.688383 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.797538 (+109155us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.801347 (+ 3809us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.814809 (+ 13462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2152101}
I20250625 07:52:41.816254 2745 tablet.cc:1722] T 4da889781dbd41aebac86e9833cf2b7e P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:41.810714 1158 tablet_bootstrap.cc:654] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.818871 1161 tablet_replica.cc:331] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.819103 1161 raft_consensus.cc:2241] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.819299 1161 raft_consensus.cc:2270] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.819813 1161 ts_tablet_manager.cc:1905] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.821487 1163 tablet_bootstrap.cc:492] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.821628 1163 ts_tablet_manager.cc:1397] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.170s user 0.021s sys 0.038s
I20250625 07:52:41.822301 1163 raft_consensus.cc:357] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.822518 1163 raft_consensus.cc:383] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.822615 1163 raft_consensus.cc:738] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.822901 1163 consensus_queue.cc:260] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.824795 1162 tablet_bootstrap.cc:492] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.824929 1162 ts_tablet_manager.cc:1397] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250625 07:52:41.825544 1162 raft_consensus.cc:357] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.825688 1162 raft_consensus.cc:383] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.825731 1162 raft_consensus.cc:738] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.825914 1162 consensus_queue.cc:260] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.826368 1162 ts_tablet_manager.cc:1428] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:41.826625 1162 tablet_bootstrap.cc:492] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.828128 1162 tablet_bootstrap.cc:654] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:41.832827 583 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38477, user_credentials={real_user=slave}} blocked reactor thread for 45668.9us
I20250625 07:52:41.834290 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 0b7daf759e324c1db853fdbaab76c721 (table default.Table_b3b5ccfb28464dcda7e964281f02f69a [id=e4963df45eee48f4a7d95d252de8cc5a]) successfully deleted
I20250625 07:52:41.824043 1163 ts_tablet_manager.cc:1428] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:41.835429 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7006ea881d43f799f84b174e421812 (DEFAULT_TABLE table=default.Table_850acc602efd46efb7b5771d5efa41f0 [id=9c21a3ebfe8d4101bf5b3620c730b6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.835811 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8f3c910a4ae34ca19663b35e08b96f41 (DEFAULT_TABLE table=default.Table_22a7b61bc1664cb8b3634346932badb7 [id=62157b121f0749b5889230a847079ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.836007 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5f863462f3684d249a732033940b489b (DEFAULT_TABLE table=default.Table_2eb4cc88dc8c42eeb3bdee821d48710d [id=4cedf96a928f424897e6d9f763647beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.836124 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7d4e51e8dd19451682fda376228fc6c9 (DEFAULT_TABLE table=default.Table_38841e5dae8e4642b89bd495dffe7174 [id=1957031d08764ffcb9be80f820d5fb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.836454 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 0b480cf26b2d46b98e029030043230d5 (DEFAULT_TABLE table=default.Table_9ac5f17525cc431e9941ba484be5bdb7 [id=e96b32ccb00e4ec3838c48597fa1782d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.836571 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 01ef509fc5fa44a6be31ebf61013caa5 (DEFAULT_TABLE table=default.Table_c122910d938045b480a31da589781477 [id=77b3796a324f4604b29c41c397d7273c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.836905 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 1b968e0d6fc04abe93245a4b7bb95ae4 (DEFAULT_TABLE table=default.Table_8a53e4ac425f4b01a2a885cf7ce1c2ca [id=b6d8e4c547874cce938ec075345826ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.838197 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0b480cf26b2d46b98e029030043230d5 (DEFAULT_TABLE table=default.Table_9ac5f17525cc431e9941ba484be5bdb7 [id=e96b32ccb00e4ec3838c48597fa1782d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.838551 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b480cf26b2d46b98e029030043230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.839857 1163 tablet_bootstrap.cc:492] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.841048 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7d4e51e8dd19451682fda376228fc6c9 (DEFAULT_TABLE table=default.Table_38841e5dae8e4642b89bd495dffe7174 [id=1957031d08764ffcb9be80f820d5fb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.841425 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4e51e8dd19451682fda376228fc6c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.841914 713 leader_election.cc:343] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.845094 887 tablet_service.cc:1468] Processing CreateTablet for tablet 01ef509fc5fa44a6be31ebf61013caa5 (DEFAULT_TABLE table=default.Table_c122910d938045b480a31da589781477 [id=77b3796a324f4604b29c41c397d7273c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.845520 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ef509fc5fa44a6be31ebf61013caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.848977 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1b968e0d6fc04abe93245a4b7bb95ae4 (DEFAULT_TABLE table=default.Table_8a53e4ac425f4b01a2a885cf7ce1c2ca [id=b6d8e4c547874cce938ec075345826ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.849391 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b968e0d6fc04abe93245a4b7bb95ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.850718 2723 raft_consensus.cc:491] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.850850 2723 raft_consensus.cc:513] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.851433 2723 leader_election.cc:290] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:41.851084 1161 ts_tablet_manager.cc:1918] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.851836 1161 log.cc:1199] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c263b671be384cd097041c60306034de
W20250625 07:52:41.849714 713 leader_election.cc:343] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c82b840442ec4cb4993636183812ed22
I20250625 07:52:41.852797 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "670f3b7f0a904656af418d01ed6f58b4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:41.853292 850 leader_election.cc:343] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.853811 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "670f3b7f0a904656af418d01ed6f58b4" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:41.853183 713 leader_election.cc:304] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.854800 1161 ts_tablet_manager.cc:1939] T c263b671be384cd097041c60306034de P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.854981 2718 raft_consensus.cc:2747] T c82b840442ec4cb4993636183812ed22 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.856362 1163 tablet_bootstrap.cc:654] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.857168 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1362) took 2319 ms. Trace:
I20250625 07:52:41.857591 1161 rpcz_store.cc:276] 0625 07:52:39.537708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.537886 (+ 178us) service_pool.cc:225] Handling call
0625 07:52:41.817594 (+2279708us) ts_tablet_manager.cc:1145] Deleting tablet c263b671be384cd097041c60306034de
0625 07:52:41.818816 (+ 1222us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.829339 (+ 10523us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.850005 (+ 20666us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.857147 (+ 7142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2279500}
W20250625 07:52:41.859660 851 leader_election.cc:343] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:41.859802 851 leader_election.cc:304] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.859750 1161 tablet_replica.cc:331] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:41.860004 2749 tablet.cc:1722] T 0352225d645c408ba8c74273ee5f56e6 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:41.860246 2723 raft_consensus.cc:2747] T 670f3b7f0a904656af418d01ed6f58b4 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.835817 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7006ea881d43f799f84b174e421812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.861166 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b480cf26b2d46b98e029030043230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.861269 1162 tablet_bootstrap.cc:492] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:41.861394 1162 ts_tablet_manager.cc:1397] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20250625 07:52:41.862079 1162 raft_consensus.cc:357] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.862238 1162 raft_consensus.cc:383] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.862284 1162 raft_consensus.cc:738] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.862455 1162 consensus_queue.cc:260] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.862880 1162 ts_tablet_manager.cc:1428] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:41.863163 1162 tablet_bootstrap.cc:492] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:41.864532 1162 tablet_bootstrap.cc:654] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.864809 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f863462f3684d249a732033940b489b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.865716 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0de0641aa91e4de682efe31fe074f8e5 consensus state has no leader
I20250625 07:52:41.866176 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3c910a4ae34ca19663b35e08b96f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.869181 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b968e0d6fc04abe93245a4b7bb95ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.872852 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ef509fc5fa44a6be31ebf61013caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.875072 1161 raft_consensus.cc:2241] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.875875 1161 raft_consensus.cc:2270] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.876479 1161 ts_tablet_manager.cc:1905] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.880575 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c263b671be384cd097041c60306034de (table default.Table_2be2bfc1127842c1aac7e83c5725e760 [id=3a68a55b7dcd4138b3903ad023cede81]) successfully deleted
I20250625 07:52:41.879312 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4e51e8dd19451682fda376228fc6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.885798 2749 tablet.cc:1722] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:41.891389 713 leader_election.cc:304] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.891654 2718 raft_consensus.cc:2802] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:41.891767 2718 raft_consensus.cc:695] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:41.892014 2718 consensus_queue.cc:237] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:41.903710 713 leader_election.cc:343] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: dddbc1200acf4e7fa93ef85b398d5832
I20250625 07:52:41.904592 713 leader_election.cc:304] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:41.912441 884 tablet_service.cc:1468] Processing CreateTablet for tablet 5f863462f3684d249a732033940b489b (DEFAULT_TABLE table=default.Table_2eb4cc88dc8c42eeb3bdee821d48710d [id=4cedf96a928f424897e6d9f763647beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.913111 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f863462f3684d249a732033940b489b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.913494 2721 raft_consensus.cc:2747] T dddbc1200acf4e7fa93ef85b398d5832 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.918097 2736 raft_consensus.cc:491] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.918295 2736 raft_consensus.cc:513] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.919100 2736 leader_election.cc:290] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:41.921276 986 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: c3d8b6b5c06248879e6052f6ae7fe46e
W20250625 07:52:41.921679 985 leader_election.cc:343] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: c3d8b6b5c06248879e6052f6ae7fe46e
I20250625 07:52:41.921772 985 leader_election.cc:304] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:41.922217 2736 raft_consensus.cc:2747] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:41.912155 2407 ts_tablet_manager.cc:1905] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.922454 1163 tablet_bootstrap.cc:492] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:41.922590 1163 ts_tablet_manager.cc:1397] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.003s
W20250625 07:52:41.923321 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5239065a44e24cf28772215a5fbab86e consensus state has no leader
I20250625 07:52:41.923288 1163 raft_consensus.cc:357] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.923446 1163 raft_consensus.cc:383] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:41.923496 1163 raft_consensus.cc:738] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:41.923481 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_44ede952c58346e98591c6e14c6ff095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.923605 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_84f144f01ac148138bdb55f29f46a4fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.924022 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44ede952c58346e98591c6e14c6ff095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:41.924237 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21d11cdcccb54af1ab68b39f9f0474a0 consensus state has no leader
I20250625 07:52:41.924536 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_125f22225a154f1288ab7309b9522d36" } new_table_name: "default.Table_7e4cff3a8bbb4119977e9e32200f6b51" modify_external_catalogs: true
I20250625 07:52:41.925462 592 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_95a6076a07824d0fa2ca5eab23b3bda2 [id=4181f3a804aa47f99e1c30e8d1307d00] alter complete (version 5)
I20250625 07:52:41.927263 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7006ea881d43f799f84b174e421812 (DEFAULT_TABLE table=default.Table_850acc602efd46efb7b5771d5efa41f0 [id=9c21a3ebfe8d4101bf5b3620c730b6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.927659 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7006ea881d43f799f84b174e421812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.929095 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ac8968da295c4b7fb6d83b84935d27d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.929491 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 363fee07295c44139fd713b33c82623c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.930528 1161 ts_tablet_manager.cc:1918] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.930693 1161 log.cc:1199] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c0cc3ba09d5e4c3d88e74fffc97eea8b
I20250625 07:52:41.931042 1161 ts_tablet_manager.cc:1939] T c0cc3ba09d5e4c3d88e74fffc97eea8b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:41.932148 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_62d807019ce448b9b92b551210892637"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.933207 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_abf60fb31fbd42b5ad3b47e078e00649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:41.933349 749 tablet_service.cc:1468] Processing CreateTablet for tablet 086b8931d4c14af08518e088dc56ad20 (DEFAULT_TABLE table=default.Table_5976d462ca1440b6b16932b369bfa86a [id=b7682b02c75e479a842995eb055d00cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:41.933689 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abf60fb31fbd42b5ad3b47e078e00649 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.933728 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086b8931d4c14af08518e088dc56ad20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.935114 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 670f3b7f0a904656af418d01ed6f58b4 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:41.935628 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6158c39c210841bc97e6b5a7c6e0b408"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:41.936040 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6158c39c210841bc97e6b5a7c6e0b408 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.936940 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1363) took 2398 ms. Trace:
I20250625 07:52:41.937033 1161 rpcz_store.cc:276] 0625 07:52:39.538238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.538405 (+ 167us) service_pool.cc:225] Handling call
0625 07:52:41.859527 (+2321122us) ts_tablet_manager.cc:1145] Deleting tablet c0cc3ba09d5e4c3d88e74fffc97eea8b
0625 07:52:41.859542 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.903889 (+ 44347us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.930419 (+ 26530us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.936918 (+ 6499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2320765}
I20250625 07:52:41.937480 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c0cc3ba09d5e4c3d88e74fffc97eea8b (table default.Table_46671c5a6dee4493836a690b730763e9 [id=81f4ba5e17054a2a8f8d53f69916f6fb]) successfully deleted
I20250625 07:52:41.938222 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1372) took 2358 ms. Trace:
I20250625 07:52:41.938302 1161 rpcz_store.cc:276] 0625 07:52:39.592217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.592478 (+ 261us) service_pool.cc:225] Handling call
0625 07:52:41.938100 (+2345622us) ts_tablet_manager.cc:1145] Deleting tablet 4dbf323dfa034cc78904cb73077f4ccf
0625 07:52:41.938111 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.938205 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2325835}
W20250625 07:52:41.938680 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4dbf323dfa034cc78904cb73077f4ccf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbf323dfa034cc78904cb73077f4ccf already in progress: creating tablet
W20250625 07:52:41.924022 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84f144f01ac148138bdb55f29f46a4fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.940232 749 tablet_service.cc:1468] Processing CreateTablet for tablet 71edb6f95ed74e1c81afe85d08638002 (DEFAULT_TABLE table=default.Table_7e5a70063453457485e565593de59bec [id=03b14f087cd64a39b8c19c978516b13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.940595 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71edb6f95ed74e1c81afe85d08638002. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:41.940786 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62d807019ce448b9b92b551210892637 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:41.942106 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_093618aa2b3243bd99e0f25c8e738c26" } modify_external_catalogs: true
I20250625 07:52:41.942265 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_093618aa2b3243bd99e0f25c8e738c26" } modify_external_catalogs: true
I20250625 07:52:41.943074 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 34de0050a09a474a86010518a40fae0f (DEFAULT_TABLE table=default.Table_4e667e326e6e46eb82ccbda12435479a [id=d278eb527bd24958a0edf6ec0dfacda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.943434 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34de0050a09a474a86010518a40fae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.944389 2407 ts_tablet_manager.cc:1918] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:41.944777 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1373) took 2352 ms. Trace:
I20250625 07:52:41.945185 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5239065a44e24cf28772215a5fbab86e (DEFAULT_TABLE table=default.Table_1b8a59feb1b443b2899089b11bdb6a2f [id=9e3863c1c2024ecbb67db85aad410b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.945361 1163 consensus_queue.cc:260] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.945518 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5239065a44e24cf28772215a5fbab86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.946188 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f3b863e17f8942c58fe854c0dab420be" } modify_external_catalogs: true
I20250625 07:52:41.946498 892 tablet_service.cc:1468] Processing CreateTablet for tablet 352ec6ddd6db42a5a98345e4c0d7e5be (DEFAULT_TABLE table=default.Table_9af73a2221464a56a5c9de529f96638e [id=12389f20eb894bb39aab6011a8ad5bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.946723 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f3b863e17f8942c58fe854c0dab420be" } modify_external_catalogs: true
I20250625 07:52:41.946909 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352ec6ddd6db42a5a98345e4c0d7e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.947649 740 tablet_service.cc:1515] Processing DeleteTablet for tablet e9363b569fb84cc0bf55e0c6e2dbc608 with delete_type TABLET_DATA_DELETED (replaced by tablet ae25051c24024b5897d8115e032ca3b0) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.948290 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7006ea881d43f799f84b174e421812 (DEFAULT_TABLE table=default.Table_850acc602efd46efb7b5771d5efa41f0 [id=9c21a3ebfe8d4101bf5b3620c730b6ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.948630 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7006ea881d43f799f84b174e421812. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.948966 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab1dd89e402429fa099dde94e1e3c68 (DEFAULT_TABLE table=default.Table_9b7eb79f055b4d6ba3bb32e5c61bdf04 [id=cd0d7c76be254bb2948625c275e3a5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.949326 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab1dd89e402429fa099dde94e1e3c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.951153 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8f3c910a4ae34ca19663b35e08b96f41 (DEFAULT_TABLE table=default.Table_22a7b61bc1664cb8b3634346932badb7 [id=62157b121f0749b5889230a847079ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.951546 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3c910a4ae34ca19663b35e08b96f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.956640 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 352ec6ddd6db42a5a98345e4c0d7e5be (DEFAULT_TABLE table=default.Table_9af73a2221464a56a5c9de529f96638e [id=12389f20eb894bb39aab6011a8ad5bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.957206 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352ec6ddd6db42a5a98345e4c0d7e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.959975 1163 ts_tablet_manager.cc:1428] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20250625 07:52:41.961601 2603 raft_consensus.cc:491] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.944849 1161 rpcz_store.cc:276] 0625 07:52:39.592355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.597767 (+ 5412us) service_pool.cc:225] Handling call
0625 07:52:41.944633 (+2346866us) ts_tablet_manager.cc:1145] Deleting tablet f51ef840a3ed40bd939b3fe62df3cda2
0625 07:52:41.944642 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.944758 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2324929}
I20250625 07:52:41.962597 1161 tablet_replica.cc:331] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:41.962590 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f51ef840a3ed40bd939b3fe62df3cda2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ef840a3ed40bd939b3fe62df3cda2 already in progress: creating tablet
I20250625 07:52:41.962798 1161 raft_consensus.cc:2241] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.962981 1161 raft_consensus.cc:2270] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.962003 741 tablet_service.cc:1468] Processing CreateTablet for tablet 5f863462f3684d249a732033940b489b (DEFAULT_TABLE table=default.Table_2eb4cc88dc8c42eeb3bdee821d48710d [id=4cedf96a928f424897e6d9f763647beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.963480 2603 raft_consensus.cc:513] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:41.963169 2407 log.cc:1199] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/fe630469fe7c4f4bac28277896eddc9c
I20250625 07:52:41.964035 2407 ts_tablet_manager.cc:1939] T fe630469fe7c4f4bac28277896eddc9c P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:41.964582 1161 ts_tablet_manager.cc:1905] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:41.963661 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f863462f3684d249a732033940b489b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.965898 754 tablet_service.cc:1468] Processing CreateTablet for tablet 7d4e51e8dd19451682fda376228fc6c9 (DEFAULT_TABLE table=default.Table_38841e5dae8e4642b89bd495dffe7174 [id=1957031d08764ffcb9be80f820d5fb12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.966133 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c858e286fbba450e81c6cc55f0f2b613 with delete_type TABLET_DATA_DELETED (replaced by tablet bedc38018eb6412b9e6ca3f0ee34cf87) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.966271 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4e51e8dd19451682fda376228fc6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.966828 752 tablet_service.cc:1515] Processing DeleteTablet for tablet dddbc1200acf4e7fa93ef85b398d5832 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.967133 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c82b840442ec4cb4993636183812ed22 with delete_type TABLET_DATA_DELETED (replaced by tablet 81cbb1bcbd7745d2bc9c4ef8c1b4bbb8) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.967419 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e181a3f1ae2c49349718d842c166b1e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.967727 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f31104720614b028e373ae1500c31cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:41.968137 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8f3c910a4ae34ca19663b35e08b96f41 (DEFAULT_TABLE table=default.Table_22a7b61bc1664cb8b3634346932badb7 [id=62157b121f0749b5889230a847079ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.968492 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f3c910a4ae34ca19663b35e08b96f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.968941 755 tablet_service.cc:1468] Processing CreateTablet for tablet 01ef509fc5fa44a6be31ebf61013caa5 (DEFAULT_TABLE table=default.Table_c122910d938045b480a31da589781477 [id=77b3796a324f4604b29c41c397d7273c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.969236 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01ef509fc5fa44a6be31ebf61013caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.972337 739 tablet_service.cc:1468] Processing CreateTablet for tablet 1b968e0d6fc04abe93245a4b7bb95ae4 (DEFAULT_TABLE table=default.Table_8a53e4ac425f4b01a2a885cf7ce1c2ca [id=b6d8e4c547874cce938ec075345826ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.972635 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b968e0d6fc04abe93245a4b7bb95ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.975389 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 26db0effa17b4cacb1450a01f9631ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:41.978293 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fed43adfae594e3b91ed1012838c3317 consensus state has no leader
I20250625 07:52:41.979579 1163 tablet_bootstrap.cc:492] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:41.982537 1163 tablet_bootstrap.cc:654] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:41.962913 894 tablet_service.cc:1468] Processing CreateTablet for tablet 34de0050a09a474a86010518a40fae0f (DEFAULT_TABLE table=default.Table_4e667e326e6e46eb82ccbda12435479a [id=d278eb527bd24958a0edf6ec0dfacda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.983852 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_69c013cd0c6a4f5397f00282c3269577" } modify_external_catalogs: true
I20250625 07:52:41.984018 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_69c013cd0c6a4f5397f00282c3269577" } modify_external_catalogs: true
I20250625 07:52:41.984185 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34de0050a09a474a86010518a40fae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.985141 2718 raft_consensus.cc:491] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:41.985258 2718 raft_consensus.cc:513] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:41.985754 2718 leader_election.cc:290] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:41.987116 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 47b4112b7e674518bcc1ea748370fd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:41.964187 2603 leader_election.cc:290] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:41.964502 740 tablet_service.cc:1468] Processing CreateTablet for tablet 0b480cf26b2d46b98e029030043230d5 (DEFAULT_TABLE table=default.Table_9ac5f17525cc431e9941ba484be5bdb7 [id=e96b32ccb00e4ec3838c48597fa1782d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.991227 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b480cf26b2d46b98e029030043230d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.992534 880 tablet_service.cc:1468] Processing CreateTablet for tablet a703a9d08d404e6f82f8090b39ba5e89 (DEFAULT_TABLE table=default.Table_4ea4205b845140cead133d2799fee68e [id=240769d1aa3547e2a9efd138407edc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.992919 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a703a9d08d404e6f82f8090b39ba5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.995074 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1453) took 2457 ms. Trace:
I20250625 07:52:41.995281 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea407cd71934b67ba0e20f5e9c4179f (DEFAULT_TABLE table=default.Table_f862fc2f9357410dae689f37371ee4d4 [id=53af66875d7b430299d354d1299dd762]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.995659 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea407cd71934b67ba0e20f5e9c4179f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.995769 2407 rpcz_store.cc:276] 0625 07:52:39.537341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.537390 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:41.766109 (+2228719us) ts_tablet_manager.cc:1145] Deleting tablet fe630469fe7c4f4bac28277896eddc9c
0625 07:52:41.766125 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.925781 (+159656us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.944313 (+ 18532us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.995053 (+ 50740us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":388,"tablet-delete.queue_time_us":2228551}
I20250625 07:52:41.996968 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea407cd71934b67ba0e20f5e9c4179f (DEFAULT_TABLE table=default.Table_f862fc2f9357410dae689f37371ee4d4 [id=53af66875d7b430299d354d1299dd762]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.997305 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea407cd71934b67ba0e20f5e9c4179f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:41.999079 2407 tablet_replica.cc:331] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:41.999277 2407 raft_consensus.cc:2241] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:41.999420 2407 raft_consensus.cc:2270] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:41.999949 2407 ts_tablet_manager.cc:1905] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.000972 876 tablet_service.cc:1468] Processing CreateTablet for tablet a3919c777a89491b8634b57cdb9dcda8 (DEFAULT_TABLE table=default.Table_32b28c84687545e0aab58d60861754c2 [id=438e1b282e8d46b6833a89f8b0fe9c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.001303 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3919c777a89491b8634b57cdb9dcda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.005384 739 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab1dd89e402429fa099dde94e1e3c68 (DEFAULT_TABLE table=default.Table_9b7eb79f055b4d6ba3bb32e5c61bdf04 [id=cd0d7c76be254bb2948625c275e3a5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.006206 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab1dd89e402429fa099dde94e1e3c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.008673 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 8fdc3220d53142faae41fd1f1500de56 (table default.Table_462e9992625a498e814ed911f3ee61a6 [id=e14d62b2aeaa4770ac1ef8ea42f2ba24]) successfully deleted
I20250625 07:52:42.013074 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0ea407cd71934b67ba0e20f5e9c4179f (DEFAULT_TABLE table=default.Table_f862fc2f9357410dae689f37371ee4d4 [id=53af66875d7b430299d354d1299dd762]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.013757 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea407cd71934b67ba0e20f5e9c4179f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.014811 740 tablet_service.cc:1468] Processing CreateTablet for tablet 352ec6ddd6db42a5a98345e4c0d7e5be (DEFAULT_TABLE table=default.Table_9af73a2221464a56a5c9de529f96638e [id=12389f20eb894bb39aab6011a8ad5bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.992063 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a3919c777a89491b8634b57cdb9dcda8 (DEFAULT_TABLE table=default.Table_32b28c84687545e0aab58d60861754c2 [id=438e1b282e8d46b6833a89f8b0fe9c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:41.992965 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 037006ded0c842a6816ba00bc29ffdd5 (DEFAULT_TABLE table=default.Table_d8490c9398a741279aa6a5633cd0387a [id=8a89cd2803544b2eb1eb2e7909066dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.015139 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352ec6ddd6db42a5a98345e4c0d7e5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.015334 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037006ded0c842a6816ba00bc29ffdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.016939 749 tablet_service.cc:1468] Processing CreateTablet for tablet a3919c777a89491b8634b57cdb9dcda8 (DEFAULT_TABLE table=default.Table_32b28c84687545e0aab58d60861754c2 [id=438e1b282e8d46b6833a89f8b0fe9c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.017242 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3919c777a89491b8634b57cdb9dcda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.019655 2603 raft_consensus.cc:491] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.019773 2603 raft_consensus.cc:513] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.019937 754 tablet_service.cc:1468] Processing CreateTablet for tablet a703a9d08d404e6f82f8090b39ba5e89 (DEFAULT_TABLE table=default.Table_4ea4205b845140cead133d2799fee68e [id=240769d1aa3547e2a9efd138407edc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.020218 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a703a9d08d404e6f82f8090b39ba5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.020360 2603 leader_election.cc:290] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.021176 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3919c777a89491b8634b57cdb9dcda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.026563 887 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab1dd89e402429fa099dde94e1e3c68 (DEFAULT_TABLE table=default.Table_9b7eb79f055b4d6ba3bb32e5c61bdf04 [id=cd0d7c76be254bb2948625c275e3a5e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.027321 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab1dd89e402429fa099dde94e1e3c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.027456 1163 tablet_bootstrap.cc:492] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.027602 1163 ts_tablet_manager.cc:1397] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20250625 07:52:42.028267 1163 raft_consensus.cc:357] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.028442 1163 raft_consensus.cc:383] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.028494 1163 raft_consensus.cc:738] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.029038 741 tablet_service.cc:1468] Processing CreateTablet for tablet 34de0050a09a474a86010518a40fae0f (DEFAULT_TABLE table=default.Table_4e667e326e6e46eb82ccbda12435479a [id=d278eb527bd24958a0edf6ec0dfacda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.029525 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34de0050a09a474a86010518a40fae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.036528 2755 tablet.cc:1722] T 0352225d645c408ba8c74273ee5f56e6 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:41.967815 1160 ts_tablet_manager.cc:1918] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.042078 1160 log.cc:1199] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/484a49fab6134d059c64f5a3f2b29e03
I20250625 07:52:42.042433 1160 ts_tablet_manager.cc:1939] T 484a49fab6134d059c64f5a3f2b29e03 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.042999 1158 tablet_bootstrap.cc:492] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.043144 1158 ts_tablet_manager.cc:1397] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.281s user 0.018s sys 0.013s
I20250625 07:52:42.043902 1158 raft_consensus.cc:357] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.044095 1158 raft_consensus.cc:383] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.044157 1158 raft_consensus.cc:738] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.044353 2755 tablet.cc:1722] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:42.046958 1163 consensus_queue.cc:260] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.044376 1158 consensus_queue.cc:260] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.050632 1158 ts_tablet_manager.cc:1428] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250625 07:52:42.050933 1158 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.051465 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90f2bef7baa54f1a96b3b59921d59457" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.053085 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1418) took 2029 ms. Trace:
I20250625 07:52:42.053978 1158 tablet_bootstrap.cc:654] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.060165 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.063041 883 tablet_service.cc:1468] Processing CreateTablet for tablet ebbda15772854f75ba45d32ca0d8de23 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.063360 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.063501 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbda15772854f75ba45d32ca0d8de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.063539 907 raft_consensus.cc:2466] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:42.063680 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet fe630469fe7c4f4bac28277896eddc9c (table default.Table_bebb528a00504a7e80d68cec36278f45 [id=75a65f2d29cb4ccdb6555b7f1d0fe16a]) successfully deleted
I20250625 07:52:42.068887 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a703a9d08d404e6f82f8090b39ba5e89 (DEFAULT_TABLE table=default.Table_4ea4205b845140cead133d2799fee68e [id=240769d1aa3547e2a9efd138407edc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.069662 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a703a9d08d404e6f82f8090b39ba5e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.058334 1163 ts_tablet_manager.cc:1428] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250625 07:52:42.061805 878 tablet_service.cc:1468] Processing CreateTablet for tablet 037006ded0c842a6816ba00bc29ffdd5 (DEFAULT_TABLE table=default.Table_d8490c9398a741279aa6a5633cd0387a [id=8a89cd2803544b2eb1eb2e7909066dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.071669 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037006ded0c842a6816ba00bc29ffdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.063035 1160 rpcz_store.cc:276] 0625 07:52:40.023209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.023382 (+ 173us) service_pool.cc:225] Handling call
0625 07:52:41.794889 (+1771507us) ts_tablet_manager.cc:1145] Deleting tablet 484a49fab6134d059c64f5a3f2b29e03
0625 07:52:41.794903 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:41.964490 (+169587us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:41.967709 (+ 3219us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.053052 (+ 85343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771217}
I20250625 07:52:42.073295 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1420) took 2049 ms. Trace:
I20250625 07:52:42.073393 1160 rpcz_store.cc:276] 0625 07:52:40.023731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.023781 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:42.072347 (+2048566us) ts_tablet_manager.cc:1145] Deleting tablet 99acd74c01f34d068c868fde3c6ec1df
0625 07:52:42.072356 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.073274 (+ 918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2048397}
I20250625 07:52:42.073805 1160 tablet_replica.cc:331] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.074044 1160 raft_consensus.cc:2241] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.074227 1160 raft_consensus.cc:2270] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.074746 1160 ts_tablet_manager.cc:1905] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.076960 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 484a49fab6134d059c64f5a3f2b29e03 (table default.Table_3e26b4b26dfe434ba65119e99f41bb8c [id=1c84ef7a8f854a9898e3d11ad9d5c675]) successfully deleted
W20250625 07:52:42.077227 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 99acd74c01f34d068c868fde3c6ec1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99acd74c01f34d068c868fde3c6ec1df already in progress: creating tablet
I20250625 07:52:42.078630 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet a7c6d068d2324ea6a07bb9be74578870 (table default.Table_2d3b375ead844126bcf8371bf41c071a [id=3ff721dec5f145209210d0ae74e38b84]). Expected version 3 got 0
I20250625 07:52:42.079254 1161 ts_tablet_manager.cc:1918] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:42.079352 1161 log.cc:1199] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/e4b87309b76f4f0e9dde120d9f755c44
I20250625 07:52:42.079392 590 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6ad13e139c64d20abeb28e8584f0781" } modify_external_catalogs: true
I20250625 07:52:42.079552 590 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_f6ad13e139c64d20abeb28e8584f0781" } modify_external_catalogs: true
I20250625 07:52:42.080542 1161 ts_tablet_manager.cc:1939] T e4b87309b76f4f0e9dde120d9f755c44 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:42.080878 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13d9c0065e34a96b6bdd29592932414 consensus state has no leader
I20250625 07:52:42.080888 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1376) took 2461 ms. Trace:
I20250625 07:52:42.080956 1161 rpcz_store.cc:276] 0625 07:52:39.619807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.619849 (+ 42us) service_pool.cc:225] Handling call
0625 07:52:41.962548 (+2342699us) ts_tablet_manager.cc:1145] Deleting tablet e4b87309b76f4f0e9dde120d9f755c44
0625 07:52:41.962558 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.034666 (+ 72108us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.079151 (+ 44485us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.080872 (+ 1721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342387}
I20250625 07:52:42.081169 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1e4d42ab40de4520ade73a572a5334dd" } modify_external_catalogs: true
I20250625 07:52:42.081302 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1e4d42ab40de4520ade73a572a5334dd" } modify_external_catalogs: true
I20250625 07:52:42.081350 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91]) successfully deleted
W20250625 07:52:42.081344 2662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:52:42.081768 2662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:52:42.081892 2662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 07:52:42.082731 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dc32deadb0e84b4abc80418d4ab5002b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.083292 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc32deadb0e84b4abc80418d4ab5002b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.084120 1012 tablet_service.cc:1468] Processing CreateTablet for tablet ebbda15772854f75ba45d32ca0d8de23 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.084677 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_37564180b46a46b7bc77bf339f37cafb" } new_table_name: "default.Table_2721d585b30249e093264064a58ddda2" modify_external_catalogs: true
I20250625 07:52:42.086272 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 21be70ccbe434a1b89ae0dac8bc4347f
I20250625 07:52:42.087386 1022 tablet_service.cc:1468] Processing CreateTablet for tablet b461009fa6ec4cf9856c735529ecb9a8 (DEFAULT_TABLE table=default.Table_332cfc4acc5744b3b4435e0460b5c13b [id=9d3c525ef1ef4c20a116d417e693825d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.088939 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.089542 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c4729a62b7434cc0bcd1eecbfe838faf
I20250625 07:52:42.090363 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.091085 2736 raft_consensus.cc:491] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.091174 884 tablet_service.cc:1468] Processing CreateTablet for tablet b461009fa6ec4cf9856c735529ecb9a8 (DEFAULT_TABLE table=default.Table_332cfc4acc5744b3b4435e0460b5c13b [id=9d3c525ef1ef4c20a116d417e693825d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.091207 2736 raft_consensus.cc:513] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.091538 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b461009fa6ec4cf9856c735529ecb9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.091770 2736 leader_election.cc:290] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:42.092788 986 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:42.096843 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:42.097760 985 leader_election.cc:343] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:42.097911 985 leader_election.cc:304] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:42.098194 2736 raft_consensus.cc:2747] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.101157 1161 tablet_replica.cc:331] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.101397 2407 ts_tablet_manager.cc:1918] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.101060 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26db0effa17b4cacb1450a01f9631ddf" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.101869 2407 log.cc:1199] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1973678168a84875b747897f08dd1bad
I20250625 07:52:42.102110 1161 raft_consensus.cc:2241] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.103845 1161 raft_consensus.cc:2270] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.104713 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.104759 2407 ts_tablet_manager.cc:1939] T 1973678168a84875b747897f08dd1bad P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.105965 2603 raft_consensus.cc:491] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.106114 2603 raft_consensus.cc:513] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.106822 2603 leader_election.cc:290] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.107705 1161 ts_tablet_manager.cc:1905] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.109907 880 tablet_service.cc:1468] Processing CreateTablet for tablet 37c563b8cd944bccbd8c2d930673ea10 (DEFAULT_TABLE table=default.Table_c1a0b0bfc2444e53bcb089f929de637f [id=0d7cca773b894fcaa7e9b8a63d12d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.110344 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c563b8cd944bccbd8c2d930673ea10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.111505 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1454) took 2573 ms. Trace:
I20250625 07:52:42.111800 2407 rpcz_store.cc:276] 0625 07:52:39.537667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.540955 (+ 3288us) service_pool.cc:225] Handling call
0625 07:52:41.999028 (+2458073us) ts_tablet_manager.cc:1145] Deleting tablet 1973678168a84875b747897f08dd1bad
0625 07:52:41.999038 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.002970 (+ 3932us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.097860 (+ 94890us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.111489 (+ 13629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457890}
I20250625 07:52:42.113658 2407 tablet_replica.cc:331] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.114101 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d3a5816969114919b085458769cc150f (DEFAULT_TABLE table=default.Table_a2c905090ebe4dc2aeeceaf4fcdbff1e [id=ea4467b20d9a46ed9f22f093f7639984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.114144 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 37c563b8cd944bccbd8c2d930673ea10 (DEFAULT_TABLE table=default.Table_c1a0b0bfc2444e53bcb089f929de637f [id=0d7cca773b894fcaa7e9b8a63d12d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.114501 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8a373acb34944e74ba0cbbd9c5aca029 (DEFAULT_TABLE table=default.Table_c83ce536556447e68e849b1839f1a5ce [id=a990b1fde4c5456a8a0d448e55414f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.114749 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a90bb52358f4c38a540ad025e9e0430 with delete_type TABLET_DATA_DELETED (replaced by tablet d0a75906385744e5ba6938f451a2d7ec) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.114144 1026 tablet_service.cc:1468] Processing CreateTablet for tablet bb1979147f1540d1b74236d007b35f53 (DEFAULT_TABLE table=default.Table_9afbd980bdec4740a7f68571c5a4fe5e [id=410c6802a1df40fc8b22817c688a4349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.115274 2407 raft_consensus.cc:2241] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.115808 876 tablet_service.cc:1468] Processing CreateTablet for tablet 8a373acb34944e74ba0cbbd9c5aca029 (DEFAULT_TABLE table=default.Table_c83ce536556447e68e849b1839f1a5ce [id=a990b1fde4c5456a8a0d448e55414f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.116178 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a373acb34944e74ba0cbbd9c5aca029. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.116103 2407 raft_consensus.cc:2270] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.117529 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f1d251171314118b66e3a921a4a50c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.118477 2407 ts_tablet_manager.cc:1905] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.119621 881 tablet_service.cc:1468] Processing CreateTablet for tablet bb1979147f1540d1b74236d007b35f53 (DEFAULT_TABLE table=default.Table_9afbd980bdec4740a7f68571c5a4fe5e [id=410c6802a1df40fc8b22817c688a4349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.119984 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1979147f1540d1b74236d007b35f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.121627 1162 tablet_bootstrap.cc:492] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.121789 1162 ts_tablet_manager.cc:1397] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.259s user 0.002s sys 0.025s
I20250625 07:52:42.122614 1162 raft_consensus.cc:357] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.122817 1162 raft_consensus.cc:383] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.122889 1162 raft_consensus.cc:738] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.123102 1162 consensus_queue.cc:260] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.123661 1162 ts_tablet_manager.cc:1428] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:42.123986 1162 tablet_bootstrap.cc:492] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.125567 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a5816969114919b085458769cc150f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.125788 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c563b8cd944bccbd8c2d930673ea10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.128875 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a373acb34944e74ba0cbbd9c5aca029. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.132040 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_323c4483e39f47e59c4b72b78dfdfc03" } new_table_name: "default.Table_f4e91d280b644abab2e435312dc3925a" modify_external_catalogs: true
I20250625 07:52:42.132216 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbda15772854f75ba45d32ca0d8de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.132927 2718 raft_consensus.cc:491] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.132966 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_63f0bd1f056c4a608b24c55877b8b80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.133054 2718 raft_consensus.cc:513] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.133517 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63f0bd1f056c4a608b24c55877b8b80a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.133611 2718 leader_election.cc:290] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.133786 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_95a02ee5862445a89fcc2bc41403f418" } new_table_name: "default.Table_e703a594772745af9cc4330052da8807" modify_external_catalogs: true
I20250625 07:52:42.135048 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1979147f1540d1b74236d007b35f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.134086 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ccf6ef165a784273982de47808303acf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.136413 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccf6ef165a784273982de47808303acf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.136816 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b461009fa6ec4cf9856c735529ecb9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.137068 2723 raft_consensus.cc:491] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.137231 2723 raft_consensus.cc:513] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.137852 2723 leader_election.cc:290] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.141685 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf9e6ee9a3c4dcd820e33ec7bcde1da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.145165 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf9e6ee9a3c4dcd820e33ec7bcde1da" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.145711 851 leader_election.cc:343] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.146792 850 leader_election.cc:343] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.146988 850 leader_election.cc:304] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.147281 2723 raft_consensus.cc:2747] T 2cf9e6ee9a3c4dcd820e33ec7bcde1da P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.155782 1161 ts_tablet_manager.cc:1918] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.155925 1161 log.cc:1199] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/7db12504a1a54add920561dae27d2d2b
I20250625 07:52:42.156201 1161 ts_tablet_manager.cc:1939] T 7db12504a1a54add920561dae27d2d2b P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.156494 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1377) took 2536 ms. Trace:
I20250625 07:52:42.160223 894 tablet_service.cc:1468] Processing CreateTablet for tablet d3a5816969114919b085458769cc150f (DEFAULT_TABLE table=default.Table_a2c905090ebe4dc2aeeceaf4fcdbff1e [id=ea4467b20d9a46ed9f22f093f7639984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.160745 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a5816969114919b085458769cc150f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.156585 1161 rpcz_store.cc:276] 0625 07:52:39.619902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.620193 (+ 291us) service_pool.cc:225] Handling call
0625 07:52:42.101115 (+2480922us) ts_tablet_manager.cc:1145] Deleting tablet 7db12504a1a54add920561dae27d2d2b
0625 07:52:42.101123 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.132662 (+ 31539us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.155681 (+ 23019us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.156479 (+ 798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2480680}
I20250625 07:52:42.170396 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1379) took 2542 ms. Trace:
I20250625 07:52:42.170502 1161 rpcz_store.cc:276] 0625 07:52:39.627663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.628509 (+ 846us) service_pool.cc:225] Handling call
0625 07:52:42.170262 (+2541753us) ts_tablet_manager.cc:1145] Deleting tablet b9eb3bcbece34f14a05dc7d95ca7ece2
0625 07:52:42.170273 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.170374 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2541514}
I20250625 07:52:42.170770 1161 tablet_replica.cc:331] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.170957 1161 raft_consensus.cc:2241] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.171123 1161 raft_consensus.cc:2270] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.171650 1161 ts_tablet_manager.cc:1905] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.177420 963 tablet.cc:2378] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:42.177526 741 tablet_service.cc:1468] Processing CreateTablet for tablet 037006ded0c842a6816ba00bc29ffdd5 (DEFAULT_TABLE table=default.Table_d8490c9398a741279aa6a5633cd0387a [id=8a89cd2803544b2eb1eb2e7909066dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.177938 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 037006ded0c842a6816ba00bc29ffdd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.180815 740 tablet_service.cc:1468] Processing CreateTablet for tablet ebbda15772854f75ba45d32ca0d8de23 (DEFAULT_TABLE table=default.Table_46be1cd72d9843de91659b93152efdd1 [id=72689041ad694029b0470f59d03affa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.181201 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbda15772854f75ba45d32ca0d8de23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.184638 2723 raft_consensus.cc:491] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.184912 2723 raft_consensus.cc:513] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.185537 2723 leader_election.cc:290] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:42.185906 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.186077 777 raft_consensus.cc:2466] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 0.
I20250625 07:52:42.186538 851 leader_election.cc:304] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:42.187290 2723 raft_consensus.cc:2802] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.187366 2723 raft_consensus.cc:491] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.187419 2723 raft_consensus.cc:3058] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.188680 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.189436 850 leader_election.cc:343] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.190479 739 tablet_service.cc:1468] Processing CreateTablet for tablet d3a5816969114919b085458769cc150f (DEFAULT_TABLE table=default.Table_a2c905090ebe4dc2aeeceaf4fcdbff1e [id=ea4467b20d9a46ed9f22f093f7639984]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.190611 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8a373acb34944e74ba0cbbd9c5aca029 (DEFAULT_TABLE table=default.Table_c83ce536556447e68e849b1839f1a5ce [id=a990b1fde4c5456a8a0d448e55414f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.190908 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a5816969114919b085458769cc150f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:42.191500 713 consensus_peers.cc:487] T 896419bd407f43e1a0a361330ff9821b P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20250625 07:52:42.191839 740 tablet_service.cc:1468] Processing CreateTablet for tablet b461009fa6ec4cf9856c735529ecb9a8 (DEFAULT_TABLE table=default.Table_332cfc4acc5744b3b4435e0460b5c13b [id=9d3c525ef1ef4c20a116d417e693825d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:42.192384 713 consensus_peers.cc:487] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:42.192922 754 tablet_service.cc:1468] Processing CreateTablet for tablet 37c563b8cd944bccbd8c2d930673ea10 (DEFAULT_TABLE table=default.Table_c1a0b0bfc2444e53bcb089f929de637f [id=0d7cca773b894fcaa7e9b8a63d12d09f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.193280 2603 raft_consensus.cc:491] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.193250 755 tablet_service.cc:1468] Processing CreateTablet for tablet bb1979147f1540d1b74236d007b35f53 (DEFAULT_TABLE table=default.Table_9afbd980bdec4740a7f68571c5a4fe5e [id=410c6802a1df40fc8b22817c688a4349]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.193374 2603 raft_consensus.cc:513] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.193892 2603 leader_election.cc:290] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.194453 2749 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:42.194962 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 7db12504a1a54add920561dae27d2d2b (table default.Table_16d107cb53e442eaa68e834ce2509d3f [id=ea8f730322a346a69e2f0098644f704d]) successfully deleted
W20250625 07:52:42.195585 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9eb3bcbece34f14a05dc7d95ca7ece2 already in progress: creating tablet
I20250625 07:52:42.199931 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 21be70ccbe434a1b89ae0dac8bc4347f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.201037 1163 tablet_bootstrap.cc:492] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.202704 2723 raft_consensus.cc:513] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.203284 2723 leader_election.cc:290] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Requested vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:42.203925 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:42.204630 850 leader_election.cc:343] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.205103 741 tablet_service.cc:1515] Processing DeleteTablet for tablet c4729a62b7434cc0bcd1eecbfe838faf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.206983 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c563b8cd944bccbd8c2d930673ea10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.212348 1163 tablet_bootstrap.cc:654] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.213142 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb1979147f1540d1b74236d007b35f53. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:42.215013 713 leader_election.cc:343] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.216248 713 leader_election.cc:343] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.216967 2749 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:42.216980 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a373acb34944e74ba0cbbd9c5aca029. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.219848 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b461009fa6ec4cf9856c735529ecb9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.224812 2723 raft_consensus.cc:491] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.225163 2723 raft_consensus.cc:513] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.225836 2723 leader_election.cc:290] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.226982 2407 ts_tablet_manager.cc:1918] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.227085 2407 log.cc:1199] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:42.227380 2407 ts_tablet_manager.cc:1939] T 3cdbdd1e3b734463ab452bafecb75822 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.228653 1158 tablet_bootstrap.cc:492] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.228819 1158 ts_tablet_manager.cc:1397] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.178s user 0.009s sys 0.004s
I20250625 07:52:42.229591 1158 raft_consensus.cc:357] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.229750 1158 raft_consensus.cc:383] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.229818 1158 raft_consensus.cc:738] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.229998 1158 consensus_queue.cc:260] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.231170 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9eaab8f86bdc45cab8645cd5a6e3489c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.231689 850 leader_election.cc:343] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.233624 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3823279d7e06435484f1b82836ebcf39
I20250625 07:52:42.234624 1162 tablet_bootstrap.cc:654] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.235622 594 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:42.236343 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_206e0491cb7c4ca49e73282736bc998d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.237689 851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:46661, user_credentials={real_user=slave}} blocked reactor thread for 51180.7us
W20250625 07:52:42.238385 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_206e0491cb7c4ca49e73282736bc998d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.238737 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34544157495f4e14a7978e0eec70b522" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7"
I20250625 07:52:42.238924 777 raft_consensus.cc:3058] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.241189 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 3823279d7e06435484f1b82836ebcf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.241609 777 raft_consensus.cc:2466] T 34544157495f4e14a7978e0eec70b522 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429026e8d98d46339db143f84897a092 in term 1.
I20250625 07:52:42.241665 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 3823279d7e06435484f1b82836ebcf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.242892 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9eaab8f86bdc45cab8645cd5a6e3489c" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.243479 851 leader_election.cc:304] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:42.243961 851 leader_election.cc:343] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.244150 851 leader_election.cc:304] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.244541 2723 raft_consensus.cc:491] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.244760 2723 raft_consensus.cc:513] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.245193 713 leader_election.cc:343] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.245519 2723 leader_election.cc:290] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.245714 2723 raft_consensus.cc:2802] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.245961 2723 raft_consensus.cc:695] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [term 1 LEADER]: Becoming Leader. State: Replica: 429026e8d98d46339db143f84897a092, State: Running, Role: LEADER
I20250625 07:52:42.246270 2723 consensus_queue.cc:237] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.248015 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23908409fbbd48dbbe82be8182ba665f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.248665 1158 ts_tablet_manager.cc:1428] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
W20250625 07:52:42.248667 850 leader_election.cc:343] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.249796 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1455) took 2708 ms. Trace:
I20250625 07:52:42.249960 2407 rpcz_store.cc:276] 0625 07:52:39.541319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.546323 (+ 5004us) service_pool.cc:225] Handling call
0625 07:52:42.113284 (+2566961us) ts_tablet_manager.cc:1145] Deleting tablet 3cdbdd1e3b734463ab452bafecb75822
0625 07:52:42.113459 (+ 175us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.136615 (+ 23156us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.226885 (+ 90270us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.249770 (+ 22885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2566783}
I20250625 07:52:42.251243 1158 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.252935 1158 tablet_bootstrap.cc:654] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.255635 590 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8cadd41655404316b10939e46d370c76
I20250625 07:52:42.256858 1162 tablet_bootstrap.cc:492] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.256958 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 0009ea8aaf6e4fdd8ff602bd59968de3
I20250625 07:52:42.256992 1162 ts_tablet_manager.cc:1397] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.133s user 0.000s sys 0.003s
I20250625 07:52:42.257040 2795 raft_consensus.cc:2747] T 9eaab8f86bdc45cab8645cd5a6e3489c P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.257372 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cadd41655404316b10939e46d370c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.258011 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:42.258051 826 tablet.cc:2378] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:42.258180 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23908409fbbd48dbbe82be8182ba665f" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.258383 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.258684 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cadd41655404316b10939e46d370c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:42.258718 851 leader_election.cc:343] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.258816 851 leader_election.cc:304] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.261433 2407 tablet_replica.cc:331] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.261616 2407 raft_consensus.cc:2241] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.261781 2407 raft_consensus.cc:2270] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:42.262300 713 leader_election.cc:343] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 90f2bef7baa54f1a96b3b59921d59457
I20250625 07:52:42.262415 713 leader_election.cc:304] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.262605 2721 raft_consensus.cc:2747] T 90f2bef7baa54f1a96b3b59921d59457 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.262732 2407 ts_tablet_manager.cc:1905] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.262770 713 leader_election.cc:304] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.262985 2721 raft_consensus.cc:2802] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.263056 2721 raft_consensus.cc:491] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.263129 2721 raft_consensus.cc:3058] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:42.265431 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65231ee1a59d49eb96ba352b74c3460c consensus state has no leader
I20250625 07:52:42.265817 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_83d5b4f023f54ac98cc89d3f2b462d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.266359 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83d5b4f023f54ac98cc89d3f2b462d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.261686 1162 raft_consensus.cc:357] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.267380 1162 raft_consensus.cc:383] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.267449 1162 raft_consensus.cc:738] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.267648 1162 consensus_queue.cc:260] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.267941 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87447ae5f35a44fe9cf5706e306bd5d7 has no consensus state
I20250625 07:52:42.268114 2795 raft_consensus.cc:2747] T 23908409fbbd48dbbe82be8182ba665f P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:42.268515 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38c221d71bff4b949596b36cb3a1ab80 has no consensus state
I20250625 07:52:42.270747 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_c7cdd85509b14915a9668f890f2ed721" } new_table_name: "default.Table_32d037a4ad4f4ce591f516a352d785e3" modify_external_catalogs: true
I20250625 07:52:42.270886 1163 tablet_bootstrap.cc:492] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.271019 1163 ts_tablet_manager.cc:1397] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250625 07:52:42.271751 1163 raft_consensus.cc:357] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.271899 1163 raft_consensus.cc:383] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.271940 1163 raft_consensus.cc:738] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.272156 1163 consensus_queue.cc:260] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.272199 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_3e4796a533d44263a8627ae547f7c8ba [id=9a2fe4250951446188a9e591f8f38b16] alter complete (version 3)
I20250625 07:52:42.272761 1163 ts_tablet_manager.cc:1428] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.273162 1163 tablet_bootstrap.cc:492] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:42.273746 713 leader_election.cc:343] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 26db0effa17b4cacb1450a01f9631ddf
I20250625 07:52:42.273880 713 leader_election.cc:304] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.274108 2612 raft_consensus.cc:2747] T 26db0effa17b4cacb1450a01f9631ddf P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.274854 1163 tablet_bootstrap.cc:654] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.276528 590 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 896419bd407f43e1a0a361330ff9821b (table default.Table_a2fb991094f545cfb3a7f7aefb03d8a5 [id=3a39e84d0aa349689ccc22a31ea282b0]). Expected version 3 got 0
I20250625 07:52:42.277292 1162 ts_tablet_manager.cc:1428] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250625 07:52:42.277737 1162 tablet_bootstrap.cc:492] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.277959 2721 raft_consensus.cc:513] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.278581 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.278770 2721 leader_election.cc:290] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.279573 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96a1f823fd3b47c9b6d1cf5647f9a68a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.280390 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d2bb8b0a8f45079083f2b42bc25011" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.283833 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:42.285122 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.285408 912 raft_consensus.cc:2466] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:42.288995 1161 ts_tablet_manager.cc:1918] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.289109 1161 log.cc:1199] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/af8c114c1c464bcc95dbd287c88976d6
I20250625 07:52:42.289479 1161 ts_tablet_manager.cc:1939] T af8c114c1c464bcc95dbd287c88976d6 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.289790 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1395) took 2508 ms. Trace:
I20250625 07:52:42.289845 2745 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:42.292567 2721 raft_consensus.cc:491] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.292748 2721 raft_consensus.cc:513] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.293411 2721 leader_election.cc:290] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.293978 1162 tablet_bootstrap.cc:654] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.294026 2785 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:42.294536 2755 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:42.295814 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cadd41655404316b10939e46d370c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.296239 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:42.296880 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.297153 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bee1c7f7746a4d6fb763aedadc861421" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:42.297308 914 raft_consensus.cc:3058] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.297554 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6c0834adf1748c6a67805d26b973171" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.300155 914 raft_consensus.cc:2466] T bee1c7f7746a4d6fb763aedadc861421 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:42.284523 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 1973678168a84875b747897f08dd1bad (table default.Table_83f5d615c53042dd888e74c415cc3caf [id=1f730d2bc75b4fab86e26a382a94ce2c]) successfully deleted
I20250625 07:52:42.294658 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5bbc315b57e741fbb6e626bd8670f92a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.303746 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbc315b57e741fbb6e626bd8670f92a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:42.304207 713 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 96a1f823fd3b47c9b6d1cf5647f9a68a
W20250625 07:52:42.304404 713 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:42.304533 713 leader_election.cc:304] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:42.295045 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_b2c2d9931ec040a6a8847d5dfa6a7f18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.305079 2612 raft_consensus.cc:2802] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.305156 2612 raft_consensus.cc:491] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.305218 2612 raft_consensus.cc:3058] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:42.305281 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2c2d9931ec040a6a8847d5dfa6a7f18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.305481 1160 ts_tablet_manager.cc:1918] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.305572 1160 log.cc:1199] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/17be34eba92246df973f3d464e13fd30
I20250625 07:52:42.305860 1160 ts_tablet_manager.cc:1939] T 17be34eba92246df973f3d464e13fd30 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.295336 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_5de26cad2ddf465da7eec8a3dbc3dd41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.306530 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5de26cad2ddf465da7eec8a3dbc3dd41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.306650 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1419) took 2283 ms. Trace:
I20250625 07:52:42.306730 1160 rpcz_store.cc:276] 0625 07:52:40.023399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.024019 (+ 620us) service_pool.cc:225] Handling call
0625 07:52:42.073754 (+2049735us) ts_tablet_manager.cc:1145] Deleting tablet 17be34eba92246df973f3d464e13fd30
0625 07:52:42.073765 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.206960 (+133195us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.305396 (+ 98436us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.306629 (+ 1233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049511}
I20250625 07:52:42.307194 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 17be34eba92246df973f3d464e13fd30 (table default.Table_6e6e1b57ae284b1d9bc03891dc267c9c [id=9e4563f37d0a4423887337f954c65aa0]) successfully deleted
I20250625 07:52:42.295679 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5bd347b575ac434b8316ddfd879697c3" } new_table_name: "default.Table_905ed787fc744bffb016385b3eb25732" modify_external_catalogs: true
I20250625 07:52:42.307564 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1421) took 2283 ms. Trace:
I20250625 07:52:42.307631 1160 rpcz_store.cc:276] 0625 07:52:40.023879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.024321 (+ 442us) service_pool.cc:225] Handling call
0625 07:52:42.307443 (+2283122us) ts_tablet_manager.cc:1145] Deleting tablet 71c66d9107024afa8b68deeac22332f6
0625 07:52:42.307453 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.307549 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2282939}
I20250625 07:52:42.307865 2612 raft_consensus.cc:513] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.295969 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ace02fdadc1a4ca086ad7557ad5fe666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.308430 2612 leader_election.cc:290] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:42.308713 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace02fdadc1a4ca086ad7557ad5fe666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.310393 713 leader_election.cc:304] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:42.310636 2721 raft_consensus.cc:2802] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.310765 2721 raft_consensus.cc:695] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:42.310988 2721 consensus_queue.cc:237] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.311143 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 3cdbdd1e3b734463ab452bafecb75822 (table default.Table_dc62a435b6d44031bc3e5a60a3914e8f [id=f24c46ab73494d2b958f0fa750eadc63]) successfully deleted
I20250625 07:52:42.313372 2745 tablet.cc:1722] T f84acd45845c4917b7dc8c5ad5232897 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250625 07:52:42.313500 2749 tablet.cc:1717] T 61e076870efc4744b1224ddcb0eef3fa P 191b611b772a4fda9d1cba98776138a7: Skipping requested alter to schema version 3, tablet already version 3
I20250625 07:52:42.318702 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 3823279d7e06435484f1b82836ebcf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:33894
W20250625 07:52:42.318876 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 71c66d9107024afa8b68deeac22332f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c66d9107024afa8b68deeac22332f6 already in progress: creating tablet
I20250625 07:52:42.319440 590 catalog_manager.cc:5582] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:42.289863 1161 rpcz_store.cc:276] 0625 07:52:39.780863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.781325 (+ 462us) service_pool.cc:225] Handling call
0625 07:52:42.170730 (+2389405us) ts_tablet_manager.cc:1145] Deleting tablet af8c114c1c464bcc95dbd287c88976d6
0625 07:52:42.170738 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.268676 (+ 97938us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.288871 (+ 20195us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.289774 (+ 903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2389129}
I20250625 07:52:42.321300 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1400) took 2267 ms. Trace:
I20250625 07:52:42.321390 1161 rpcz_store.cc:276] 0625 07:52:40.053895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.053949 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:42.321173 (+2267224us) ts_tablet_manager.cc:1145] Deleting tablet 99acd74c01f34d068c868fde3c6ec1df
0625 07:52:42.321185 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.321281 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267051}
I20250625 07:52:42.322336 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d1c96cdddd3443b8b67f57f4d6c4846e (table default.Table_802e0f6519e044c0a5b420289568590f [id=1082e4c0af594912bd6b02bd57557cf0]). Expected version 1 got 0
I20250625 07:52:42.320996 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet af8c114c1c464bcc95dbd287c88976d6 (table default.Table_72e90a4803a04ea09a7b37b41bc7b65a [id=765fd58a211e4d4e923e1a25c113702f]) successfully deleted
W20250625 07:52:42.324890 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 99acd74c01f34d068c868fde3c6ec1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99acd74c01f34d068c868fde3c6ec1df already in progress: creating tablet
W20250625 07:52:42.325125 713 leader_election.cc:343] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.325394 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1399) took 2271 ms. Trace:
I20250625 07:52:42.325824 1163 tablet_bootstrap.cc:492] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.325959 1163 ts_tablet_manager.cc:1397] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250625 07:52:42.325958 1161 rpcz_store.cc:276] 0625 07:52:40.053799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.054082 (+ 283us) service_pool.cc:225] Handling call
0625 07:52:42.325211 (+2271129us) ts_tablet_manager.cc:1145] Deleting tablet 71c66d9107024afa8b68deeac22332f6
0625 07:52:42.325220 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.325374 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270976}
W20250625 07:52:42.326457 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 71c66d9107024afa8b68deeac22332f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c66d9107024afa8b68deeac22332f6 already in progress: creating tablet
W20250625 07:52:42.326519 713 leader_election.cc:343] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.326661 713 leader_election.cc:304] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.326782 1163 raft_consensus.cc:357] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.327024 1163 raft_consensus.cc:383] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.327111 1163 raft_consensus.cc:738] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.327410 2612 raft_consensus.cc:2747] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.327495 1163 consensus_queue.cc:260] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.327921 713 leader_election.cc:343] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.328054 713 leader_election.cc:304] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:42.328230 713 raft_consensus.cc:2684] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250625 07:52:42.329243 1163 ts_tablet_manager.cc:1428] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:42.328913 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1401) took 2152 ms. Trace:
I20250625 07:52:42.329885 1163 tablet_bootstrap.cc:492] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.329694 1161 rpcz_store.cc:276] 0625 07:52:40.176878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.176931 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:42.328790 (+2151859us) ts_tablet_manager.cc:1145] Deleting tablet ba49ad0cb12d4f738288aea215dbda87
0625 07:52:42.328804 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.328896 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149581}
W20250625 07:52:42.330832 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ba49ad0cb12d4f738288aea215dbda87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba49ad0cb12d4f738288aea215dbda87 already in progress: creating tablet
I20250625 07:52:42.331254 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1402) took 2154 ms. Trace:
I20250625 07:52:42.332088 1163 tablet_bootstrap.cc:654] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.332194 1161 rpcz_store.cc:276] 0625 07:52:40.176999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.177149 (+ 150us) service_pool.cc:225] Handling call
0625 07:52:42.331132 (+2153983us) ts_tablet_manager.cc:1145] Deleting tablet 7b3eafac1b2b46acb2ac6fca087b7f16
0625 07:52:42.331144 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.331237 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153772}
W20250625 07:52:42.332679 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b3eafac1b2b46acb2ac6fca087b7f16 already in progress: creating tablet
I20250625 07:52:42.333134 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1403) took 2156 ms. Trace:
I20250625 07:52:42.333670 1162 tablet_bootstrap.cc:492] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.333809 1162 ts_tablet_manager.cc:1397] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.002s
I20250625 07:52:42.334463 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1422) took 2167 ms. Trace:
I20250625 07:52:42.333789 1161 rpcz_store.cc:276] 0625 07:52:40.177089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.177156 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:42.333020 (+2155864us) ts_tablet_manager.cc:1145] Deleting tablet b4d68f1d9d8244a2b49f5e1a5f2855f2
0625 07:52:42.333029 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.333118 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155625}
I20250625 07:52:42.334550 1160 rpcz_store.cc:276] 0625 07:52:40.166644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.166711 (+ 67us) service_pool.cc:225] Handling call
0625 07:52:42.334321 (+2167610us) ts_tablet_manager.cc:1145] Deleting tablet ba49ad0cb12d4f738288aea215dbda87
0625 07:52:42.334331 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.334443 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167344}
I20250625 07:52:42.334903 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1423) took 2168 ms. Trace:
I20250625 07:52:42.335000 1160 rpcz_store.cc:276] 0625 07:52:40.166758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.166807 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:42.334768 (+2167961us) ts_tablet_manager.cc:1145] Deleting tablet 7b3eafac1b2b46acb2ac6fca087b7f16
0625 07:52:42.334777 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.334888 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167780}
W20250625 07:52:42.335085 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 already in progress: creating tablet
I20250625 07:52:42.335260 1160 tablet_replica.cc:331] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.335446 1160 raft_consensus.cc:2241] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.335611 1160 raft_consensus.cc:2270] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:42.335614 713 leader_election.cc:343] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.335927 713 leader_election.cc:343] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.336052 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1404) took 2107 ms. Trace:
I20250625 07:52:42.336130 1160 ts_tablet_manager.cc:1905] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.337061 1162 raft_consensus.cc:357] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.337229 1162 raft_consensus.cc:383] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.337280 1162 raft_consensus.cc:738] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.336166 1161 rpcz_store.cc:276] 0625 07:52:40.228222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.228291 (+ 69us) service_pool.cc:225] Handling call
0625 07:52:42.335450 (+2107159us) ts_tablet_manager.cc:1145] Deleting tablet cd226579c77b4938bb031fac0412cbb1
0625 07:52:42.335461 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.335648 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106935}
I20250625 07:52:42.337464 1162 consensus_queue.cc:260] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.338492 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1405) took 2096 ms. Trace:
I20250625 07:52:42.337934 1162 ts_tablet_manager.cc:1428] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:42.338616 1161 rpcz_store.cc:276] 0625 07:52:40.241921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.241993 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:42.338393 (+2096400us) ts_tablet_manager.cc:1145] Deleting tablet f64c42861e60451e92131a435f95b012
0625 07:52:42.338402 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.338476 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096132}
W20250625 07:52:42.339277 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f64c42861e60451e92131a435f95b012 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f64c42861e60451e92131a435f95b012 already in progress: creating tablet
I20250625 07:52:42.339269 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:42.339322 1162 tablet_bootstrap.cc:492] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.339721 1161 tablet_replica.cc:331] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:42.339948 713 leader_election.cc:343] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.340147 1161 raft_consensus.cc:2241] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.340833 1161 raft_consensus.cc:2270] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.341432 1161 ts_tablet_manager.cc:1905] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.341831 1162 tablet_bootstrap.cc:654] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.354029 1158 tablet_bootstrap.cc:492] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.354190 1158 ts_tablet_manager.cc:1397] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20250625 07:52:42.354943 1158 raft_consensus.cc:357] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.355147 1158 raft_consensus.cc:383] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.355211 1158 raft_consensus.cc:738] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.355423 1158 consensus_queue.cc:260] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.357335 2407 ts_tablet_manager.cc:1918] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.357517 2407 log.cc:1199] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:42.357931 2407 ts_tablet_manager.cc:1939] T 26d2bb8b0a8f45079083f2b42bc25011 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.358032 1158 ts_tablet_manager.cc:1428] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250625 07:52:42.358376 1158 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.359009 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1456) took 2812 ms. Trace:
I20250625 07:52:42.359149 2407 rpcz_store.cc:276] 0625 07:52:39.546656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.546697 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:42.261374 (+2714677us) ts_tablet_manager.cc:1145] Deleting tablet 26d2bb8b0a8f45079083f2b42bc25011
0625 07:52:42.261386 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.266038 (+ 4652us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.357215 (+ 91177us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.358976 (+ 1761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2703578}
I20250625 07:52:42.359565 2407 tablet_replica.cc:331] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.359874 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 26d2bb8b0a8f45079083f2b42bc25011 (table default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]) successfully deleted
I20250625 07:52:42.371708 1158 tablet_bootstrap.cc:654] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.377514 1163 tablet_bootstrap.cc:492] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.378038 1163 ts_tablet_manager.cc:1397] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250625 07:52:42.384894 2407 raft_consensus.cc:2241] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.385162 2407 raft_consensus.cc:2270] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.385773 2407 ts_tablet_manager.cc:1905] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.387998 1163 raft_consensus.cc:357] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.388892 1161 ts_tablet_manager.cc:1918] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.389048 1163 raft_consensus.cc:383] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.389173 1163 raft_consensus.cc:738] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.389307 1161 log.cc:1199] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/716e0c4dab584788bbb7631b4ca934bf
I20250625 07:52:42.390288 1161 ts_tablet_manager.cc:1939] T 716e0c4dab584788bbb7631b4ca934bf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.390588 2721 raft_consensus.cc:491] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.390751 2721 raft_consensus.cc:513] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.391448 2721 leader_election.cc:290] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.392206 1162 tablet_bootstrap.cc:492] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.392329 1162 ts_tablet_manager.cc:1397] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.001s
I20250625 07:52:42.393098 1162 raft_consensus.cc:357] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.393256 1162 raft_consensus.cc:383] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.393319 1162 raft_consensus.cc:738] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.393507 1162 consensus_queue.cc:260] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.393998 2723 raft_consensus.cc:491] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.394170 1162 ts_tablet_manager.cc:1428] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.394130 2723 raft_consensus.cc:513] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.394712 2723 leader_election.cc:290] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.394901 1162 tablet_bootstrap.cc:492] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.395750 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1c150fb54c405c8dc61ac133d98b13" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.396629 851 leader_election.cc:343] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.396900 1162 tablet_bootstrap.cc:654] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.389910 1163 consensus_queue.cc:260] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.392371 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.401013 2662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:52:42.401180 2662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:52:42.401322 2662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:52:42.401861 2662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:52:42.402043 1163 ts_tablet_manager.cc:1428] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250625 07:52:42.402110 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_88dd1fc436b94436a71598f9b420faf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.402233 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1406) took 2156 ms. Trace:
I20250625 07:52:42.402311 1161 rpcz_store.cc:276] 0625 07:52:40.246058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.246117 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:42.339678 (+2093561us) ts_tablet_manager.cc:1145] Deleting tablet 716e0c4dab584788bbb7631b4ca934bf
0625 07:52:42.339687 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.373341 (+ 33654us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.388778 (+ 15437us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.402216 (+ 13438us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2093344}
I20250625 07:52:42.402593 1161 tablet_replica.cc:331] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:42.402700 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88dd1fc436b94436a71598f9b420faf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.402822 1161 raft_consensus.cc:2241] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.402962 1161 raft_consensus.cc:2270] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.403441 1161 ts_tablet_manager.cc:1905] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.404017 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3449de0badda4189b81bdad909bd368d consensus state has no leader
I20250625 07:52:42.404417 596 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_97bafcae580545d58042a9e2bee40c93 [id=9c2f61bc035a4a26a4c079902b04259a] alter complete (version 2)
I20250625 07:52:42.405031 1163 tablet_bootstrap.cc:492] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.404963 596 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bc09c435e0cd40328ca47a8a9b9c6c7c" } modify_external_catalogs: true
I20250625 07:52:42.405115 596 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bc09c435e0cd40328ca47a8a9b9c6c7c" } modify_external_catalogs: true
I20250625 07:52:42.406591 1163 tablet_bootstrap.cc:654] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.407006 1160 ts_tablet_manager.cc:1918] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.407110 1160 log.cc:1199] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/35a93e9198d1417bb1b8e00560921483
I20250625 07:52:42.407393 1160 ts_tablet_manager.cc:1939] T 35a93e9198d1417bb1b8e00560921483 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.408084 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1424) took 2241 ms. Trace:
I20250625 07:52:42.408157 1160 rpcz_store.cc:276] 0625 07:52:40.167108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.167327 (+ 219us) service_pool.cc:225] Handling call
0625 07:52:42.335214 (+2167887us) ts_tablet_manager.cc:1145] Deleting tablet 35a93e9198d1417bb1b8e00560921483
0625 07:52:42.335223 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.373999 (+ 38776us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.406887 (+ 32888us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.408066 (+ 1179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167739}
I20250625 07:52:42.408432 1160 tablet_replica.cc:331] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.408591 1160 raft_consensus.cc:2241] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.408752 1160 raft_consensus.cc:2270] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.409229 1160 ts_tablet_manager.cc:1905] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.409420 713 leader_election.cc:343] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.409870 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 716e0c4dab584788bbb7631b4ca934bf (table default.Table_a1adb2364ae94c0aa62bc2d84fc9aded [id=5a02a39814b54e2faec76f24dc897208]) successfully deleted
I20250625 07:52:42.410786 593 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b975ec5e79ac42bfaa61d193c8ac170e" } modify_external_catalogs: true
I20250625 07:52:42.410943 593 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b975ec5e79ac42bfaa61d193c8ac170e" } modify_external_catalogs: true
I20250625 07:52:42.392458 1158 tablet_bootstrap.cc:492] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.411478 1158 ts_tablet_manager.cc:1397] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.010s
I20250625 07:52:42.412292 1158 raft_consensus.cc:357] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.412444 1158 raft_consensus.cc:383] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.412495 1158 raft_consensus.cc:738] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.415274 1158 consensus_queue.cc:260] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.416659 1158 ts_tablet_manager.cc:1428] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:42.418252 1158 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:42.418375 713 leader_election.cc:343] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a6c0834adf1748c6a67805d26b973171
I20250625 07:52:42.418502 713 leader_election.cc:304] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.418891 2721 raft_consensus.cc:2747] T a6c0834adf1748c6a67805d26b973171 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.419699 2749 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:42.421181 2799 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:42.422355 1158 tablet_bootstrap.cc:654] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.427110 1163 tablet_bootstrap.cc:492] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.427254 1163 ts_tablet_manager.cc:1397] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250625 07:52:42.428256 2407 ts_tablet_manager.cc:1918] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.428360 2407 log.cc:1199] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/96a1f823fd3b47c9b6d1cf5647f9a68a
I20250625 07:52:42.428359 2736 raft_consensus.cc:491] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.428490 2736 raft_consensus.cc:513] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.428625 2407 ts_tablet_manager.cc:1939] T 96a1f823fd3b47c9b6d1cf5647f9a68a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.429104 2736 leader_election.cc:290] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:42.429672 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ba49ad0cb12d4f738288aea215dbda87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba49ad0cb12d4f738288aea215dbda87 already in progress: creating tablet
W20250625 07:52:42.432621 985 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:42.432833 2736 raft_consensus.cc:491] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:42.432863 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b3eafac1b2b46acb2ac6fca087b7f16 already in progress: creating tablet
I20250625 07:52:42.432964 2736 raft_consensus.cc:513] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.433516 2736 leader_election.cc:290] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.433506 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 35a93e9198d1417bb1b8e00560921483 (table default.Table_deb382f48ba949ed979b457ddbd62fc1 [id=bf668cc9e9db4f529d2badcae5254439]) successfully deleted
I20250625 07:52:42.434247 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.435323 1163 raft_consensus.cc:357] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.435540 1163 raft_consensus.cc:383] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.435624 1163 raft_consensus.cc:738] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.435883 1163 consensus_queue.cc:260] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.437237 1163 ts_tablet_manager.cc:1428] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 07:52:42.438220 1163 tablet_bootstrap.cc:492] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.439881 1163 tablet_bootstrap.cc:654] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.441109 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7d2d9da2b28f4e77a4369967441acd13" } modify_external_catalogs: true
I20250625 07:52:42.441368 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7d2d9da2b28f4e77a4369967441acd13" } modify_external_catalogs: true
I20250625 07:52:42.442040 1161 ts_tablet_manager.cc:1918] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.442147 1161 log.cc:1199] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9cc1842f257d409ba0a56873f582f023
I20250625 07:52:42.442431 1161 ts_tablet_manager.cc:1939] T 9cc1842f257d409ba0a56873f582f023 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.442723 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1407) took 2156 ms. Trace:
I20250625 07:52:42.442800 1161 rpcz_store.cc:276] 0625 07:52:40.286522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.286571 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:42.402551 (+2115980us) ts_tablet_manager.cc:1145] Deleting tablet 9cc1842f257d409ba0a56873f582f023
0625 07:52:42.402559 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.420526 (+ 17967us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.441912 (+ 21386us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.442707 (+ 795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2115784}
I20250625 07:52:42.443205 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1408) took 2055 ms. Trace:
I20250625 07:52:42.443275 1161 rpcz_store.cc:276] 0625 07:52:40.388101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.388171 (+ 70us) service_pool.cc:225] Handling call
0625 07:52:42.443079 (+2054908us) ts_tablet_manager.cc:1145] Deleting tablet d320734990404d1e9222bea1d88a18fe
0625 07:52:42.443087 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.443190 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2054683}
I20250625 07:52:42.443588 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1411) took 1882 ms. Trace:
I20250625 07:52:42.443646 1161 rpcz_store.cc:276] 0625 07:52:40.561050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.561108 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:42.443484 (+1882376us) ts_tablet_manager.cc:1145] Deleting tablet 1bbf1cfac4e14a20b3fb53607cdf100f
0625 07:52:42.443491 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.443576 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1882194}
I20250625 07:52:42.443956 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1414) took 1882 ms. Trace:
I20250625 07:52:42.444028 1161 rpcz_store.cc:276] 0625 07:52:40.561600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.563677 (+ 2077us) service_pool.cc:225] Handling call
0625 07:52:42.443844 (+1880167us) ts_tablet_manager.cc:1145] Deleting tablet 6d6b4bca32de4a538071a652409e93b2
0625 07:52:42.443852 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.443940 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880003}
I20250625 07:52:42.444315 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1445) took 1785 ms. Trace:
I20250625 07:52:42.444379 1161 rpcz_store.cc:276] 0625 07:52:40.658786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.667406 (+ 8620us) service_pool.cc:225] Handling call
0625 07:52:42.444220 (+1776814us) ts_tablet_manager.cc:1145] Deleting tablet a642a958550a40a48453f47d01b062a2
0625 07:52:42.444227 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.444303 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725840}
I20250625 07:52:42.444591 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1c150fb54c405c8dc61ac133d98b13" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.445485 1158 tablet_bootstrap.cc:492] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.445640 1158 ts_tablet_manager.cc:1397] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20250625 07:52:42.446502 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9cc1842f257d409ba0a56873f582f023 (table default.Table_3bfd98b3f05b4d5a88364b2dac9b7d1c [id=d96bccb4baa248b3bc8bfd8df41af575]) successfully deleted
I20250625 07:52:42.446584 1158 raft_consensus.cc:357] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.446702 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d320734990404d1e9222bea1d88a18fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d320734990404d1e9222bea1d88a18fe already in progress: creating tablet
I20250625 07:52:42.446816 1158 raft_consensus.cc:383] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.446871 1158 raft_consensus.cc:738] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:42.446892 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 1bbf1cfac4e14a20b3fb53607cdf100f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bbf1cfac4e14a20b3fb53607cdf100f already in progress: creating tablet
W20250625 07:52:42.447027 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6d6b4bca32de4a538071a652409e93b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6b4bca32de4a538071a652409e93b2 already in progress: creating tablet
W20250625 07:52:42.447197 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a642a958550a40a48453f47d01b062a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a642a958550a40a48453f47d01b062a2 already in progress: creating tablet
I20250625 07:52:42.447134 1158 consensus_queue.cc:260] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.448084 1158 ts_tablet_manager.cc:1428] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:42.448175 1162 tablet_bootstrap.cc:492] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.448318 1162 ts_tablet_manager.cc:1397] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250625 07:52:42.449044 1162 raft_consensus.cc:357] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.449209 1162 raft_consensus.cc:383] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.449265 1162 raft_consensus.cc:738] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.449481 1162 consensus_queue.cc:260] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.451148 1162 ts_tablet_manager.cc:1428] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250625 07:52:42.452780 2723 raft_consensus.cc:491] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.452906 894 tablet_service.cc:1515] Processing DeleteTablet for tablet ec1e58fe6d72451682bd47d7238560b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:38 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.452929 2723 raft_consensus.cc:513] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.453508 2723 leader_election.cc:290] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.456111 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.456296 912 raft_consensus.cc:2466] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:42.457084 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1c96cdddd3443b8b67f57f4d6c4846e" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:42.457238 910 raft_consensus.cc:3058] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.459996 910 raft_consensus.cc:2466] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:42.460366 2785 tablet.cc:1717] T 61e076870efc4744b1224ddcb0eef3fa P 429026e8d98d46339db143f84897a092: Skipping requested alter to schema version 3, tablet already version 3
I20250625 07:52:42.461410 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57c3d0810cf44a33b2acd50707931d26" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.462047 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "896419bd407f43e1a0a361330ff9821b" candidate_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.462473 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57c3d0810cf44a33b2acd50707931d26" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.462477 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1389e17927584ad697c9c92802a0aed7" } modify_external_catalogs: true
I20250625 07:52:42.462687 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1389e17927584ad697c9c92802a0aed7" } modify_external_catalogs: true
I20250625 07:52:42.463187 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d6fdf8a3b2db409cb38d372aab234798" } modify_external_catalogs: true
I20250625 07:52:42.463317 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d6fdf8a3b2db409cb38d372aab234798" } modify_external_catalogs: true
W20250625 07:52:42.465041 851 leader_election.cc:343] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.465327 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1457) took 2918 ms. Trace:
I20250625 07:52:42.465412 2407 rpcz_store.cc:276] 0625 07:52:39.546938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.547008 (+ 70us) service_pool.cc:225] Handling call
0625 07:52:42.359516 (+2812508us) ts_tablet_manager.cc:1145] Deleting tablet 96a1f823fd3b47c9b6d1cf5647f9a68a
0625 07:52:42.359525 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.425176 (+ 65651us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.428168 (+ 2992us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.465305 (+ 37137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2812391}
I20250625 07:52:42.465895 2407 tablet_replica.cc:331] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.466085 2407 raft_consensus.cc:2241] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.466230 2407 raft_consensus.cc:2270] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.466296 986 leader_election.cc:304] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
W20250625 07:52:42.466706 986 leader_election.cc:343] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:42.466899 986 leader_election.cc:304] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:42.467458 2736 raft_consensus.cc:2747] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.467617 2736 raft_consensus.cc:2747] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.474396 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9c021cd55d3e4dd5aacc7954f40aff2e" } new_table_name: "default.Table_aac458c1d0584b6ba421e82323079db5" modify_external_catalogs: true
W20250625 07:52:42.475359 1098 tablet.cc:2378] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:52:42.475824 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a615798650df4adf8c4304f18d3b3262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.476287 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a615798650df4adf8c4304f18d3b3262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.479353 2721 raft_consensus.cc:491] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.479518 2721 raft_consensus.cc:513] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.480187 2721 leader_election.cc:290] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.480441 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1446) took 1821 ms. Trace:
I20250625 07:52:42.480527 1161 rpcz_store.cc:276] 0625 07:52:40.658883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.674150 (+ 15267us) service_pool.cc:225] Handling call
0625 07:52:42.444559 (+1770409us) ts_tablet_manager.cc:1145] Deleting tablet a571983779854eb398b3b0b42c34f83a
0625 07:52:42.450836 (+ 6277us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.480421 (+ 29585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723783}
I20250625 07:52:42.480963 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1450) took 1591 ms. Trace:
I20250625 07:52:42.481045 1161 rpcz_store.cc:276] 0625 07:52:40.889710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.889768 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:42.480840 (+1591072us) ts_tablet_manager.cc:1145] Deleting tablet e15c7ac8ff934a8ca40ac480c7c352ab
0625 07:52:42.480852 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.480947 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1590844}
I20250625 07:52:42.481361 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1449) took 1592 ms. Trace:
I20250625 07:52:42.481431 1161 rpcz_store.cc:276] 0625 07:52:40.889373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.890072 (+ 699us) service_pool.cc:225] Handling call
0625 07:52:42.481258 (+1591186us) ts_tablet_manager.cc:1145] Deleting tablet fd6c3a490883477dbafcb74d39be7345
0625 07:52:42.481266 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.481347 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1590998}
I20250625 07:52:42.481669 1161 tablet_replica.cc:331] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.481559 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_082ee08aca1e46f0adba4610a6b4ff4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.481865 1161 raft_consensus.cc:2241] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.482021 1161 raft_consensus.cc:2270] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:42.482069 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_082ee08aca1e46f0adba4610a6b4ff4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.482329 713 leader_election.cc:304] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.482550 1161 ts_tablet_manager.cc:1905] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.482689 2721 raft_consensus.cc:2802] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.482754 2721 raft_consensus.cc:491] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.482812 2721 raft_consensus.cc:3058] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.483250 1158 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.485167 1158 tablet_bootstrap.cc:654] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.485517 2721 raft_consensus.cc:513] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.486390 2721 leader_election.cc:290] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.487408 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:42.487561 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17be34eba92246df973f3d464e13fd30" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.487859 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1
I20250625 07:52:42.488317 1163 tablet_bootstrap.cc:492] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.488437 1163 ts_tablet_manager.cc:1397] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.005s
I20250625 07:52:42.488828 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_772868a19c60415e8d8d18220e78ab27" } new_table_name: "default.Table_6ee2a87e06f046c5a004f63d525e42f3" modify_external_catalogs: true
I20250625 07:52:42.489169 1163 raft_consensus.cc:357] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.489312 1163 raft_consensus.cc:383] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.489353 1163 raft_consensus.cc:738] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.489519 1163 consensus_queue.cc:260] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.490952 1163 ts_tablet_manager.cc:1428] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 07:52:42.492398 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a571983779854eb398b3b0b42c34f83a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a571983779854eb398b3b0b42c34f83a already in progress: creating tablet
I20250625 07:52:42.492486 1163 tablet_bootstrap.cc:492] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:42.492619 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e15c7ac8ff934a8ca40ac480c7c352ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e15c7ac8ff934a8ca40ac480c7c352ab already in progress: creating tablet
W20250625 07:52:42.492802 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fd6c3a490883477dbafcb74d39be7345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd6c3a490883477dbafcb74d39be7345 already in progress: creating tablet
I20250625 07:52:42.492859 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:42.493211 850 leader_election.cc:343] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.493317 850 leader_election.cc:304] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:42.493335 713 leader_election.cc:343] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.493513 713 leader_election.cc:343] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.493860 713 leader_election.cc:304] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.494112 2603 raft_consensus.cc:2802] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.494210 2723 raft_consensus.cc:2747] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.494230 2603 raft_consensus.cc:695] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:42.494540 2603 consensus_queue.cc:237] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.497301 1163 tablet_bootstrap.cc:654] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:42.498538 850 leader_election.cc:343] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.498626 850 leader_election.cc:304] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.498813 2723 raft_consensus.cc:2747] T 57c3d0810cf44a33b2acd50707931d26 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.502090 1161 ts_tablet_manager.cc:1918] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.502211 1161 log.cc:1199] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/da895c90d6ee40379fe8181c2dbd7a1e
I20250625 07:52:42.502519 1161 ts_tablet_manager.cc:1939] T da895c90d6ee40379fe8181c2dbd7a1e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.502858 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1448) took 1633 ms. Trace:
I20250625 07:52:42.502954 1161 rpcz_store.cc:276] 0625 07:52:40.869068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.962069 (+ 93001us) service_pool.cc:225] Handling call
0625 07:52:42.481627 (+1519558us) ts_tablet_manager.cc:1145] Deleting tablet da895c90d6ee40379fe8181c2dbd7a1e
0625 07:52:42.481635 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.485932 (+ 4297us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.501971 (+ 16039us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.502839 (+ 868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519131}
I20250625 07:52:42.503607 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet da895c90d6ee40379fe8181c2dbd7a1e (table default.Table_444c92729289455eaf1a979360c97e5e [id=6fec84e2fefb4a738bce54d790e73177]) successfully deleted
I20250625 07:52:42.504331 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.505097 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d930a09f5e542aab943a51072f0c626" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:42.505278 914 raft_consensus.cc:3058] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.508041 914 raft_consensus.cc:2466] T 7d930a09f5e542aab943a51072f0c626 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:42.508505 2407 ts_tablet_manager.cc:1905] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.509706 1162 tablet_bootstrap.cc:492] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.511395 2785 tablet.cc:1722] T 15766b87fb1b4620970d5fab21465cc4 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:42.511734 1162 tablet_bootstrap.cc:654] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:42.516367 713 leader_election.cc:343] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 17be34eba92246df973f3d464e13fd30
I20250625 07:52:42.516522 713 leader_election.cc:304] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.516789 2721 raft_consensus.cc:2747] T 17be34eba92246df973f3d464e13fd30 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.517127 713 leader_election.cc:304] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.517405 2721 raft_consensus.cc:2802] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.517438 1163 tablet_bootstrap.cc:492] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.517529 2721 raft_consensus.cc:695] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:42.517663 1163 ts_tablet_manager.cc:1397] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20250625 07:52:42.517769 2721 consensus_queue.cc:237] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.518442 1163 raft_consensus.cc:357] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.518602 1163 raft_consensus.cc:383] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.518672 1163 raft_consensus.cc:738] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.518870 1163 consensus_queue.cc:260] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.519501 1163 ts_tablet_manager.cc:1428] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.519799 1163 tablet_bootstrap.cc:492] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.521304 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1452) took 1538 ms. Trace:
I20250625 07:52:42.521407 1161 rpcz_store.cc:276] 0625 07:52:40.982696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.982867 (+ 171us) service_pool.cc:225] Handling call
0625 07:52:42.521132 (+1538265us) ts_tablet_manager.cc:1145] Deleting tablet 670974e43d7e4886bfe5eb3cf36ab6ce
0625 07:52:42.521145 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.521277 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538050}
I20250625 07:52:42.522017 1160 ts_tablet_manager.cc:1918] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.522135 1160 log.cc:1199] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/47c9aca996884ffeab88381dd3cb189b
I20250625 07:52:42.522429 1160 ts_tablet_manager.cc:1939] T 47c9aca996884ffeab88381dd3cb189b P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:42.522364 584 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:43317, user_credentials={real_user=slave}} blocked reactor thread for 59079.8us
I20250625 07:52:42.522707 1163 tablet_bootstrap.cc:654] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.523073 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 90f2bef7baa54f1a96b3b59921d59457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.523466 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a6c0834adf1748c6a67805d26b973171 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.523456 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 96a1f823fd3b47c9b6d1cf5647f9a68a (table default.Table_7fc614d790ab4feaa0e3167c5de7cffd [id=93204c5397784d4fa49cd180005e4d5b]) successfully deleted
I20250625 07:52:42.523782 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a40dacaa394747aaa100a0527e14eed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.523932 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1451) took 1541 ms. Trace:
I20250625 07:52:42.524108 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.524034 1161 rpcz_store.cc:276] 0625 07:52:40.982255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.983128 (+ 873us) service_pool.cc:225] Handling call
0625 07:52:42.523787 (+1540659us) ts_tablet_manager.cc:1145] Deleting tablet d43d98c0c51b40c6bdb0c2705465038e
0625 07:52:42.523800 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.523910 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540531}
W20250625 07:52:42.524439 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 670974e43d7e4886bfe5eb3cf36ab6ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670974e43d7e4886bfe5eb3cf36ab6ce already in progress: creating tablet
W20250625 07:52:42.524679 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet d43d98c0c51b40c6bdb0c2705465038e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43d98c0c51b40c6bdb0c2705465038e already in progress: creating tablet
I20250625 07:52:42.525148 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1453) took 1463 ms. Trace:
I20250625 07:52:42.525224 1161 rpcz_store.cc:276] 0625 07:52:41.062196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.062255 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:42.525035 (+1462780us) ts_tablet_manager.cc:1145] Deleting tablet 9b83f4c8544946d9adb3346df200b215
0625 07:52:42.525044 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.525130 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462579}
W20250625 07:52:42.525599 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 9b83f4c8544946d9adb3346df200b215 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b83f4c8544946d9adb3346df200b215 already in progress: creating tablet
I20250625 07:52:42.525897 1161 tablet_replica.cc:331] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.526082 1161 raft_consensus.cc:2241] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.526228 1161 raft_consensus.cc:2270] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.526705 1161 ts_tablet_manager.cc:1905] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.527761 1162 tablet_bootstrap.cc:492] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.527889 1162 ts_tablet_manager.cc:1397] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250625 07:52:42.528316 1158 tablet_bootstrap.cc:492] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.528461 1158 ts_tablet_manager.cc:1397] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20250625 07:52:42.528532 1162 raft_consensus.cc:357] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.528710 1162 raft_consensus.cc:383] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.528769 1162 raft_consensus.cc:738] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.529065 1162 consensus_queue.cc:260] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.529273 1158 raft_consensus.cc:357] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.529448 1158 raft_consensus.cc:383] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.529510 1158 raft_consensus.cc:738] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.529515 1162 ts_tablet_manager.cc:1428] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.529702 1158 consensus_queue.cc:260] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.529943 1162 tablet_bootstrap.cc:492] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.530200 1158 ts_tablet_manager.cc:1428] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250625 07:52:42.530483 1158 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.531785 1162 tablet_bootstrap.cc:654] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:42.537886 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f08cb1194a5540aeb4ec5f35089d00a7 consensus state has no leader
I20250625 07:52:42.541119 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_6861a0fca1944b48ae1ad106573ae13f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.541697 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6861a0fca1944b48ae1ad106573ae13f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.543130 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_250afd2ed99c43b4a24721d178e5e372" } new_table_name: "default.Table_c30b6032c08743888c13c120d548a6a2" modify_external_catalogs: true
W20250625 07:52:42.544337 589 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c84eab61f1254a67a528c314cd9ab899 consensus state has no leader
I20250625 07:52:42.544565 589 master_service.cc:948] ReplaceTablet: received request to replace tablet bedc38018eb6412b9e6ca3f0ee34cf87 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:42.549034 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1426) took 2381 ms. Trace:
I20250625 07:52:42.550698 1160 rpcz_store.cc:276] 0625 07:52:40.167290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.167534 (+ 244us) service_pool.cc:225] Handling call
0625 07:52:42.408389 (+2240855us) ts_tablet_manager.cc:1145] Deleting tablet 47c9aca996884ffeab88381dd3cb189b
0625 07:52:42.408399 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.486859 (+ 78460us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.521901 (+ 35042us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.549007 (+ 27106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2240730}
I20250625 07:52:42.551082 1160 tablet_replica.cc:331] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.551288 1160 raft_consensus.cc:2241] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.551445 1160 raft_consensus.cc:2270] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.551788 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_97d0bd83faf6445c9cfcc7eca28145d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.551960 1160 ts_tablet_manager.cc:1905] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.552377 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97d0bd83faf6445c9cfcc7eca28145d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.560938 1163 tablet_bootstrap.cc:492] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.561081 1163 ts_tablet_manager.cc:1397] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250625 07:52:42.561816 1163 raft_consensus.cc:357] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.561996 1163 raft_consensus.cc:383] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.562047 1163 raft_consensus.cc:738] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.562269 1163 consensus_queue.cc:260] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.563688 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 47c9aca996884ffeab88381dd3cb189b (table default.Table_7e8eb7e9baf5497ca408e7592b57e4dc [id=53a8409531d94cc7aa5c8ecb4f8e1894]) successfully deleted
I20250625 07:52:42.567137 1163 ts_tablet_manager.cc:1428] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250625 07:52:42.567632 1163 tablet_bootstrap.cc:492] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.568845 2407 ts_tablet_manager.cc:1918] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.568975 2407 log.cc:1199] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/ea62dfb8f20f4caeaa79ea1e85d99930
I20250625 07:52:42.569264 2407 ts_tablet_manager.cc:1939] T ea62dfb8f20f4caeaa79ea1e85d99930 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.570132 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1458) took 3022 ms. Trace:
I20250625 07:52:42.570226 2407 rpcz_store.cc:276] 0625 07:52:39.547237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.547274 (+ 37us) service_pool.cc:225] Handling call
0625 07:52:42.465848 (+2918574us) ts_tablet_manager.cc:1145] Deleting tablet ea62dfb8f20f4caeaa79ea1e85d99930
0625 07:52:42.465857 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.545933 (+ 80076us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.568720 (+ 22787us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.570110 (+ 1390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2918446}
I20250625 07:52:42.570608 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1460) took 3013 ms. Trace:
I20250625 07:52:42.570778 1163 tablet_bootstrap.cc:654] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.570685 2407 rpcz_store.cc:276] 0625 07:52:39.556780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.556834 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:42.570484 (+3013650us) ts_tablet_manager.cc:1145] Deleting tablet 35f819ef1fa74982a50d6714de305535
0625 07:52:42.570493 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.570591 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3013461}
I20250625 07:52:42.571036 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1469) took 2922 ms. Trace:
I20250625 07:52:42.571117 2407 rpcz_store.cc:276] 0625 07:52:39.648344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.648573 (+ 229us) service_pool.cc:225] Handling call
0625 07:52:42.570904 (+2922331us) ts_tablet_manager.cc:1145] Deleting tablet 4dbf323dfa034cc78904cb73077f4ccf
0625 07:52:42.570912 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.571019 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2922015}
I20250625 07:52:42.571359 2407 tablet_replica.cc:331] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.571530 2407 raft_consensus.cc:2241] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:52:42.571882 2407 raft_consensus.cc:2270] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.572428 2407 ts_tablet_manager.cc:1905] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.572540 1161 ts_tablet_manager.cc:1918] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.572639 1161 log.cc:1199] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8bc8f47541c54c4287ebb7b892daa1be
I20250625 07:52:42.573027 1161 ts_tablet_manager.cc:1939] T 8bc8f47541c54c4287ebb7b892daa1be P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.573148 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet ea62dfb8f20f4caeaa79ea1e85d99930 (table default.Table_d375845276c6425baf08c8e94b318afc [id=74a388127a4343ff94b26af12d3d1577]) successfully deleted
I20250625 07:52:42.573343 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1454) took 1489 ms. Trace:
W20250625 07:52:42.573515 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 35f819ef1fa74982a50d6714de305535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35f819ef1fa74982a50d6714de305535 already in progress: creating tablet
I20250625 07:52:42.573434 1161 rpcz_store.cc:276] 0625 07:52:41.083507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.083568 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:42.525854 (+1442286us) ts_tablet_manager.cc:1145] Deleting tablet 8bc8f47541c54c4287ebb7b892daa1be
0625 07:52:42.525862 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.556749 (+ 30887us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.572411 (+ 15662us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.573325 (+ 914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442087}
I20250625 07:52:42.573580 2721 raft_consensus.cc:491] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.573704 2721 raft_consensus.cc:513] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.573943 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1455) took 1336 ms. Trace:
W20250625 07:52:42.573948 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 4dbf323dfa034cc78904cb73077f4ccf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbf323dfa034cc78904cb73077f4ccf already in progress: creating tablet
I20250625 07:52:42.574021 1161 rpcz_store.cc:276] 0625 07:52:41.237429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.237494 (+ 65us) service_pool.cc:225] Handling call
0625 07:52:42.573830 (+1336336us) ts_tablet_manager.cc:1145] Deleting tablet f2909c29db4c434c9db393c12ea48a1d
0625 07:52:42.573841 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.573929 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336131}
I20250625 07:52:42.574323 2721 leader_election.cc:290] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.574446 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1456) took 1245 ms. Trace:
I20250625 07:52:42.574579 1161 rpcz_store.cc:276] 0625 07:52:41.328632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.328722 (+ 90us) service_pool.cc:225] Handling call
0625 07:52:42.574348 (+1245626us) ts_tablet_manager.cc:1145] Deleting tablet b3a84f7d816148788725216f713b3980
0625 07:52:42.574357 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.574432 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245389}
I20250625 07:52:42.574976 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1457) took 1245 ms. Trace:
I20250625 07:52:42.575058 1161 rpcz_store.cc:276] 0625 07:52:41.329008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.329065 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:42.574886 (+1245821us) ts_tablet_manager.cc:1145] Deleting tablet dba8780a44864169ac9cd63ab2e89fe9
0625 07:52:42.574894 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.574963 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245642}
I20250625 07:52:42.575531 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1458) took 1187 ms. Trace:
I20250625 07:52:42.575716 1161 rpcz_store.cc:276] 0625 07:52:41.387942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.387992 (+ 50us) service_pool.cc:225] Handling call
0625 07:52:42.575366 (+1187374us) ts_tablet_manager.cc:1145] Deleting tablet e609e7267a66453dab2b666d69aebd02
0625 07:52:42.575375 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.575515 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187156}
I20250625 07:52:42.575825 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40dacaa394747aaa100a0527e14eed0" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.575979 1161 tablet_replica.cc:331] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:42.576416 713 leader_election.cc:343] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.576167 1161 raft_consensus.cc:2241] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.577039 1161 raft_consensus.cc:2270] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.577734 1161 ts_tablet_manager.cc:1905] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.582782 713 leader_election.cc:343] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: a40dacaa394747aaa100a0527e14eed0
I20250625 07:52:42.582967 2723 raft_consensus.cc:491] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.582998 713 leader_election.cc:304] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.583128 2723 raft_consensus.cc:513] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.583487 2603 raft_consensus.cc:2747] T a40dacaa394747aaa100a0527e14eed0 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.583719 2723 leader_election.cc:290] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.581511 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8bc8f47541c54c4287ebb7b892daa1be (table default.Table_ea8eb0c6e92048c482cd9244519292c9 [id=7436f8604f214cf19d159874a3b1e48b]) successfully deleted
W20250625 07:52:42.584152 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet f2909c29db4c434c9db393c12ea48a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2909c29db4c434c9db393c12ea48a1d already in progress: creating tablet
I20250625 07:52:42.582755 1162 tablet_bootstrap.cc:492] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:42.584363 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet b3a84f7d816148788725216f713b3980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a84f7d816148788725216f713b3980 already in progress: creating tablet
I20250625 07:52:42.584391 1162 ts_tablet_manager.cc:1397] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
W20250625 07:52:42.584525 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet dba8780a44864169ac9cd63ab2e89fe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba8780a44864169ac9cd63ab2e89fe9 already in progress: creating tablet
W20250625 07:52:42.584741 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e609e7267a66453dab2b666d69aebd02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e609e7267a66453dab2b666d69aebd02 already in progress: creating tablet
I20250625 07:52:42.584715 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "daf115dd3aa548dc802ed0362c274d53" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.585372 851 leader_election.cc:343] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.585439 1162 raft_consensus.cc:357] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.586071 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "daf115dd3aa548dc802ed0362c274d53" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.585669 1162 raft_consensus.cc:383] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.588655 1162 raft_consensus.cc:738] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.591203 1162 consensus_queue.cc:260] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.592099 1162 ts_tablet_manager.cc:1428] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:42.594456 1161 ts_tablet_manager.cc:1918] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.594565 1161 log.cc:1199] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c3d8b6b5c06248879e6052f6ae7fe46e
I20250625 07:52:42.594846 1161 ts_tablet_manager.cc:1939] T c3d8b6b5c06248879e6052f6ae7fe46e P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.595824 2723 raft_consensus.cc:491] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.595965 2723 raft_consensus.cc:513] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.596499 2723 leader_election.cc:290] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.597200 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9336068493fe4245b4365f5acfdacf77" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.598099 851 leader_election.cc:343] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.598467 850 leader_election.cc:343] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.598762 850 leader_election.cc:304] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.599190 2723 raft_consensus.cc:2747] T daf115dd3aa548dc802ed0362c274d53 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.597203 1162 tablet_bootstrap.cc:492] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.600296 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1459) took 1045 ms. Trace:
I20250625 07:52:42.600332 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9336068493fe4245b4365f5acfdacf77" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.600391 1161 rpcz_store.cc:276] 0625 07:52:41.555059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.555305 (+ 246us) service_pool.cc:225] Handling call
0625 07:52:42.575940 (+1020635us) ts_tablet_manager.cc:1145] Deleting tablet c3d8b6b5c06248879e6052f6ae7fe46e
0625 07:52:42.575949 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.587823 (+ 11874us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.594340 (+ 6517us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.600278 (+ 5938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020427}
I20250625 07:52:42.600919 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c3d8b6b5c06248879e6052f6ae7fe46e (table default.Table_525806c470a840e8b54ce05f09aaf215 [id=5f0cb87ee23c4e6ea5ea57ff5b665a95]) successfully deleted
I20250625 07:52:42.601780 1161 tablet_replica.cc:331] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.601847 1162 tablet_bootstrap.cc:654] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.601991 1161 raft_consensus.cc:2241] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:42.602586 850 leader_election.cc:343] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.603266 850 leader_election.cc:304] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.607115 2723 raft_consensus.cc:2747] T 9336068493fe4245b4365f5acfdacf77 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.609048 1160 ts_tablet_manager.cc:1918] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.609282 1160 log.cc:1199] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/e06ecad6c7464ffe8ec6f632634307c9
I20250625 07:52:42.609743 1160 ts_tablet_manager.cc:1939] T e06ecad6c7464ffe8ec6f632634307c9 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.611629 2662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38477,127.0.104.61:43317,127.0.104.60:33457
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=34053
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:22 UTC on 24a791456cd2
build id 6760
ASAN enabled
I20250625 07:52:42.611845 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a927841594164e699cb248584b750caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.612443 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a927841594164e699cb248584b750caf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:42.612803 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:36221, user_credentials={real_user=slave}} blocked reactor thread for 44527.1us
I20250625 07:52:42.613366 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eba8e86294da42e99c34521b94c9978e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.613855 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eba8e86294da42e99c34521b94c9978e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.614223 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_483af3005b49459e94b99b27f58ceb1d" } new_table_name: "default.Table_7480147b60fe4768a694da67d0648a06" modify_external_catalogs: true
I20250625 07:52:42.614809 2662 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:52:42.615537 2723 raft_consensus.cc:491] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.615670 2723 raft_consensus.cc:513] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.616294 2723 leader_election.cc:290] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.616966 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c00ad0924e234c9d889b4ef7c2048ddb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.617542 851 leader_election.cc:343] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.617770 1163 tablet_bootstrap.cc:492] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.617889 1163 ts_tablet_manager.cc:1397] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.008s
I20250625 07:52:42.618162 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c00ad0924e234c9d889b4ef7c2048ddb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.618682 850 leader_election.cc:343] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.618943 850 leader_election.cc:304] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.619179 2723 raft_consensus.cc:2747] T c00ad0924e234c9d889b4ef7c2048ddb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.620429 1163 raft_consensus.cc:357] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.620586 1163 raft_consensus.cc:383] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.620635 1163 raft_consensus.cc:738] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.620832 1161 raft_consensus.cc:2270] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.620903 2749 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:42.621536 2785 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:42.626051 1161 ts_tablet_manager.cc:1905] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.627678 1163 consensus_queue.cc:260] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.628538 1163 ts_tablet_manager.cc:1428] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250625 07:52:42.628886 1163 tablet_bootstrap.cc:492] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.630694 2799 tablet.cc:1722] T d4c744fef0e24cb39c4bb0787adb1cbc P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:42.631414 1163 tablet_bootstrap.cc:654] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:42.632355 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69f0d45815ee45f3b48e4675f7ed901f consensus state has no leader
I20250625 07:52:42.633304 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_feadef2afe4a46b6beb77ab7bba6c789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.633888 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feadef2afe4a46b6beb77ab7bba6c789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.635025 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1425) took 2467 ms. Trace:
I20250625 07:52:42.635160 1160 rpcz_store.cc:276] 0625 07:52:40.167200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.167809 (+ 609us) service_pool.cc:225] Handling call
0625 07:52:42.551029 (+2383220us) ts_tablet_manager.cc:1145] Deleting tablet e06ecad6c7464ffe8ec6f632634307c9
0625 07:52:42.551041 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.555241 (+ 4200us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.608893 (+ 53652us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.635005 (+ 26112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2383099}
I20250625 07:52:42.637161 1161 ts_tablet_manager.cc:1918] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.637271 1161 log.cc:1199] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/5179a144a9a0445f933ca02e79bbc122
I20250625 07:52:42.637574 1161 ts_tablet_manager.cc:1939] T 5179a144a9a0445f933ca02e79bbc122 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.637692 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1427) took 2469 ms. Trace:
I20250625 07:52:42.637883 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1461) took 1060 ms. Trace:
I20250625 07:52:42.637959 1161 rpcz_store.cc:276] 0625 07:52:41.577024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.577092 (+ 68us) service_pool.cc:225] Handling call
0625 07:52:42.601707 (+1024615us) ts_tablet_manager.cc:1145] Deleting tablet 5179a144a9a0445f933ca02e79bbc122
0625 07:52:42.601735 (+ 28us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.633778 (+ 32043us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.637073 (+ 3295us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.637870 (+ 797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1024377}
I20250625 07:52:42.638273 1161 tablet_replica.cc:331] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.638412 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 5179a144a9a0445f933ca02e79bbc122 (table default.Table_aacfa5f603c8417288db11b6ae09f880 [id=27704cd5a6534ec5bc97c7b774fc4a01]) successfully deleted
I20250625 07:52:42.638450 1161 raft_consensus.cc:2241] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.638589 1161 raft_consensus.cc:2270] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.638852 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet e06ecad6c7464ffe8ec6f632634307c9 (table default.Table_2207905ba3a74490b97826054d872e6c [id=76f35e31204549ddba67624bd1813977]) successfully deleted
I20250625 07:52:42.638789 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_800d3bfac311452cb5344f647c46cfb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.639106 1161 ts_tablet_manager.cc:1905] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.639314 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_800d3bfac311452cb5344f647c46cfb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.639846 1162 tablet_bootstrap.cc:492] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.639986 1162 ts_tablet_manager.cc:1397] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250625 07:52:42.640661 1162 raft_consensus.cc:357] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.640856 1162 raft_consensus.cc:383] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.640937 1162 raft_consensus.cc:738] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.641158 1162 consensus_queue.cc:260] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.641546 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dcc596eca41947289921078947e54512" } new_table_name: "default.Table_52c15150adb34323927aadb8acbc21ab" modify_external_catalogs: true
I20250625 07:52:42.642351 1162 ts_tablet_manager.cc:1428] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.642927 1162 tablet_bootstrap.cc:492] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.644866 2603 raft_consensus.cc:491] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.645097 1162 tablet_bootstrap.cc:654] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.645020 2603 raft_consensus.cc:513] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.645762 2603 leader_election.cc:290] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.646780 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.646952 909 raft_consensus.cc:2466] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:42.650594 713 leader_election.cc:304] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters:
I20250625 07:52:42.650936 2603 raft_consensus.cc:2802] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.651072 2603 raft_consensus.cc:491] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.651157 2603 raft_consensus.cc:3058] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.651101 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.654356 713 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.653777 2603 raft_consensus.cc:513] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.655560 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
W20250625 07:52:42.656134 713 leader_election.cc:343] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.655786 2603 leader_election.cc:290] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.646672 1160 rpcz_store.cc:276] 0625 07:52:40.167776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.167971 (+ 195us) service_pool.cc:225] Handling call
0625 07:52:42.637466 (+2469495us) ts_tablet_manager.cc:1145] Deleting tablet b4d68f1d9d8244a2b49f5e1a5f2855f2
0625 07:52:42.637478 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.637670 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2469364}
I20250625 07:52:42.656783 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea2b581d722d4a8283005b37106f5b51" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:42.656953 909 raft_consensus.cc:3058] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.657047 1160 tablet_replica.cc:331] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.657239 1160 raft_consensus.cc:2241] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.657369 1160 raft_consensus.cc:2270] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.657843 1160 ts_tablet_manager.cc:1905] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.659540 909 raft_consensus.cc:2466] T ea2b581d722d4a8283005b37106f5b51 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:42.660751 1161 ts_tablet_manager.cc:1918] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250625 07:52:42.660956 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4d68f1d9d8244a2b49f5e1a5f2855f2 already in progress: creating tablet
I20250625 07:52:42.661096 713 leader_election.cc:304] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.661633 2603 raft_consensus.cc:2802] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.661695 1161 log.cc:1199] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/a41ffd566593499299aba5513ccb2f30
I20250625 07:52:42.662097 2603 raft_consensus.cc:695] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:42.662524 1161 ts_tablet_manager.cc:1939] T a41ffd566593499299aba5513ccb2f30 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.663031 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1460) took 1086 ms. Trace:
I20250625 07:52:42.662987 2603 consensus_queue.cc:237] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.663630 1161 rpcz_store.cc:276] 0625 07:52:41.576900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.577219 (+ 319us) service_pool.cc:225] Handling call
0625 07:52:42.638225 (+1061006us) ts_tablet_manager.cc:1145] Deleting tablet a41ffd566593499299aba5513ccb2f30
0625 07:52:42.638236 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.642066 (+ 3830us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.660582 (+ 18516us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.663016 (+ 2434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1060815}
I20250625 07:52:42.661899 1158 tablet_bootstrap.cc:654] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.665287 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet a41ffd566593499299aba5513ccb2f30 (table default.Table_91ecb03f4d35473da357098c9dd96216 [id=16459cf0283447e9a445cf210eaa88f5]) successfully deleted
I20250625 07:52:42.665922 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1464) took 1007 ms. Trace:
I20250625 07:52:42.666342 1161 rpcz_store.cc:276] 0625 07:52:41.658152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.658215 (+ 63us) service_pool.cc:225] Handling call
0625 07:52:42.665794 (+1007579us) ts_tablet_manager.cc:1145] Deleting tablet 487705c4b9ae4ab097771a69f03f05dc
0625 07:52:42.665805 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.665905 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1007342}
I20250625 07:52:42.666813 2407 ts_tablet_manager.cc:1918] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:42.666957 2407 log.cc:1199] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e4b87309b76f4f0e9dde120d9f755c44
W20250625 07:52:42.667068 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 487705c4b9ae4ab097771a69f03f05dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 487705c4b9ae4ab097771a69f03f05dc already in progress: creating tablet
I20250625 07:52:42.667327 2407 ts_tablet_manager.cc:1939] T e4b87309b76f4f0e9dde120d9f755c44 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:42.667973 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet ac8968da295c4b7fb6d83b84935d27d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac8968da295c4b7fb6d83b84935d27d2 already in progress: creating tablet
I20250625 07:52:42.668162 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1473) took 3003 ms. Trace:
I20250625 07:52:42.668275 2407 rpcz_store.cc:276] 0625 07:52:39.664618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.664661 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:42.571315 (+2906654us) ts_tablet_manager.cc:1145] Deleting tablet e4b87309b76f4f0e9dde120d9f755c44
0625 07:52:42.571327 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.632812 (+ 61485us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.666667 (+ 33855us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.668108 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2906444}
I20250625 07:52:42.668723 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e4b87309b76f4f0e9dde120d9f755c44 (table default.Table_9a780105a1f44d1f8ed934b8a1ee6b32 [id=45c3c59676b0417fb034a9ffd06fee91]) successfully deleted
I20250625 07:52:42.668826 2723 raft_consensus.cc:491] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:42.668987 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4ee3563a2d994e6e88d2e0715a7b3816 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ee3563a2d994e6e88d2e0715a7b3816 already in progress: creating tablet
I20250625 07:52:42.668975 2723 raft_consensus.cc:513] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.669628 2723 leader_election.cc:290] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:42.669703 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet acfea07d8441415e92d59f2f4e0b72a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acfea07d8441415e92d59f2f4e0b72a6 already in progress: creating tablet
I20250625 07:52:42.670270 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cdee1535e5a4a21804e28e78aaea0eb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.670423 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 565d312783f24b9b995595f413ce4697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565d312783f24b9b995595f413ce4697 already in progress: creating tablet
W20250625 07:52:42.670820 851 leader_election.cc:343] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.671682 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cdee1535e5a4a21804e28e78aaea0eb" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.672196 850 leader_election.cc:343] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.672336 850 leader_election.cc:304] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.672569 2723 raft_consensus.cc:2747] T 0cdee1535e5a4a21804e28e78aaea0eb P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.673609 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1ed3b38a61bf4d4698d32b2a1e257ca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.673907 2662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250625 07:52:42.677197 1161 tablet_replica.cc:331] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.677983 1161 raft_consensus.cc:2241] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.678723 1161 raft_consensus.cc:2270] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.681826 1161 ts_tablet_manager.cc:1905] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:42.675539 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 363fee07295c44139fd713b33c82623c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 363fee07295c44139fd713b33c82623c already in progress: creating tablet
W20250625 07:52:42.683807 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ed3b38a61bf4d4698d32b2a1e257ca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:42.684144 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet e9363b569fb84cc0bf55e0c6e2dbc608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9363b569fb84cc0bf55e0c6e2dbc608 already in progress: creating tablet
I20250625 07:52:42.686399 2811 raft_consensus.cc:491] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.686522 2811 raft_consensus.cc:513] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:42.686645 589 master_service.cc:948] ReplaceTablet: received request to replace tablet a8f25febb41c403ca8be268206157756 from {username='slave'} at 127.0.0.1:34788
W20250625 07:52:42.687083 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 749849e1bb19426c97effb4b815e42ad has no consensus state
I20250625 07:52:42.687124 2811 leader_election.cc:290] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:42.688591 986 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9a90bb52358f4c38a540ad025e9e0430
W20250625 07:52:42.689042 985 leader_election.cc:343] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:42.689131 985 leader_election.cc:304] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:42.689610 2811 raft_consensus.cc:2747] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.692178 1160 ts_tablet_manager.cc:1918] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.692674 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1475) took 3027 ms. Trace:
I20250625 07:52:42.693229 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_660b719818b5402aad468695b624707c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.693774 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_660b719818b5402aad468695b624707c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.699323 1161 ts_tablet_manager.cc:1918] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250625 07:52:42.699452 1161 log.cc:1199] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/c4729a62b7434cc0bcd1eecbfe838faf
I20250625 07:52:42.699780 1161 ts_tablet_manager.cc:1939] T c4729a62b7434cc0bcd1eecbfe838faf P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.700317 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet c4729a62b7434cc0bcd1eecbfe838faf (table default.Table_093618aa2b3243bd99e0f25c8e738c26 [id=a4988551a0cf4953b4991b6c9ecc498d]) successfully deleted
I20250625 07:52:42.700497 2407 rpcz_store.cc:276] 0625 07:52:39.665100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.680304 (+ 15204us) service_pool.cc:225] Handling call
0625 07:52:42.692437 (+3012133us) ts_tablet_manager.cc:1145] Deleting tablet b9eb3bcbece34f14a05dc7d95ca7ece2
0625 07:52:42.692455 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.692595 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011925}
W20250625 07:52:42.701000 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet b9eb3bcbece34f14a05dc7d95ca7ece2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9eb3bcbece34f14a05dc7d95ca7ece2 already in progress: creating tablet
I20250625 07:52:42.701807 1162 tablet_bootstrap.cc:492] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
W20250625 07:52:42.701967 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 21be70ccbe434a1b89ae0dac8bc4347f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21be70ccbe434a1b89ae0dac8bc4347f already in progress: creating tablet
I20250625 07:52:42.702020 1162 ts_tablet_manager.cc:1397] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250625 07:52:42.702287 1161 tablet_replica.cc:331] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.702812 1162 raft_consensus.cc:357] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.703032 1162 raft_consensus.cc:383] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.703104 1162 raft_consensus.cc:738] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.703341 1162 consensus_queue.cc:260] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.704140 1162 ts_tablet_manager.cc:1428] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:42.704447 1162 tablet_bootstrap.cc:492] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.705348 1158 tablet_bootstrap.cc:492] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.705533 1158 ts_tablet_manager.cc:1397] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.175s user 0.022s sys 0.000s
I20250625 07:52:42.706247 1158 raft_consensus.cc:357] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.706418 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1470) took 3057 ms. Trace:
I20250625 07:52:42.706549 1158 raft_consensus.cc:383] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.706523 2407 rpcz_store.cc:276] 0625 07:52:39.648446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:39.800881 (+152435us) service_pool.cc:225] Handling call
0625 07:52:42.701088 (+2900207us) ts_tablet_manager.cc:1145] Deleting tablet f51ef840a3ed40bd939b3fe62df3cda2
0625 07:52:42.701100 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.706358 (+ 5258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2848307}
I20250625 07:52:42.706647 1158 raft_consensus.cc:738] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
W20250625 07:52:42.706984 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet f51ef840a3ed40bd939b3fe62df3cda2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f51ef840a3ed40bd939b3fe62df3cda2 already in progress: creating tablet
I20250625 07:52:42.707402 1158 consensus_queue.cc:260] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.707953 2407 tablet_replica.cc:331] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.708169 2407 raft_consensus.cc:2241] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.708338 2407 raft_consensus.cc:2270] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.708899 2407 ts_tablet_manager.cc:1905] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.710719 1162 tablet_bootstrap.cc:654] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.710652 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ee665ce3235a4226b7bca8e30b6ef86f" } modify_external_catalogs: true
I20250625 07:52:42.710824 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ee665ce3235a4226b7bca8e30b6ef86f" } modify_external_catalogs: true
I20250625 07:52:42.716307 1158 ts_tablet_manager.cc:1428] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250625 07:52:42.718252 1161 raft_consensus.cc:2241] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.718441 1161 raft_consensus.cc:2270] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.718823 1158 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.700338 2723 raft_consensus.cc:491] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.720865 2723 raft_consensus.cc:513] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.701422 1163 tablet_bootstrap.cc:492] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.721426 1163 ts_tablet_manager.cc:1397] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.011s
I20250625 07:52:42.721552 2723 leader_election.cc:290] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.722167 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47b4112b7e674518bcc1ea748370fd85" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.722204 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47b4112b7e674518bcc1ea748370fd85" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.722687 850 leader_election.cc:343] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.722798 851 leader_election.cc:343] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.722894 851 leader_election.cc:304] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.722872 1160 log.cc:1199] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/506794253c84486faa926de885c48c58
I20250625 07:52:42.723354 1160 ts_tablet_manager.cc:1939] T 506794253c84486faa926de885c48c58 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.723495 2723 raft_consensus.cc:2747] T 47b4112b7e674518bcc1ea748370fd85 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.724148 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1428) took 2546 ms. Trace:
I20250625 07:52:42.724278 1160 rpcz_store.cc:276] 0625 07:52:40.178046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.178105 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:42.656999 (+2478894us) ts_tablet_manager.cc:1145] Deleting tablet 506794253c84486faa926de885c48c58
0625 07:52:42.657009 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.689039 (+ 32030us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.692088 (+ 3049us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.724127 (+ 32039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2478691}
I20250625 07:52:42.724818 1160 tablet_replica.cc:331] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.725059 1160 raft_consensus.cc:2241] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.725257 1160 raft_consensus.cc:2270] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.725821 1160 ts_tablet_manager.cc:1905] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.726117 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 506794253c84486faa926de885c48c58 (table default.Table_35586b691c9a48068784a8147f80b6ef [id=a3fa61e4bf994ec2b82fa32ede09fa0f]) successfully deleted
I20250625 07:52:42.727149 1163 raft_consensus.cc:357] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.727305 1163 raft_consensus.cc:383] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.727356 1163 raft_consensus.cc:738] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.727551 1163 consensus_queue.cc:260] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.728427 1158 tablet_bootstrap.cc:654] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.729209 1163 ts_tablet_manager.cc:1428] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:42.729642 1161 ts_tablet_manager.cc:1905] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.731367 1163 tablet_bootstrap.cc:492] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.733068 1163 tablet_bootstrap.cc:654] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.735086 1162 tablet_bootstrap.cc:492] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.735249 1162 ts_tablet_manager.cc:1397] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250625 07:52:42.736104 1162 raft_consensus.cc:357] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.736323 1162 raft_consensus.cc:383] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.736423 1162 raft_consensus.cc:738] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:42.737506 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 749849e1bb19426c97effb4b815e42ad (table default.Table_52c15150adb34323927aadb8acbc21ab [id=079b5c1c6894499eb8278ef1a9883f19], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 749849e1bb19426c97effb4b815e42ad has no consensus state
I20250625 07:52:42.740885 1162 consensus_queue.cc:260] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.741241 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3b5acb6fb6cd40d2855892e2d28bed37" } new_table_name: "default.Table_52312b12583e44d7ab55e40afdaa8433" modify_external_catalogs: true
I20250625 07:52:42.744385 1162 ts_tablet_manager.cc:1428] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250625 07:52:42.745564 1162 tablet_bootstrap.cc:492] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.747294 1162 tablet_bootstrap.cc:654] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.747290 2603 consensus_queue.cc:1035] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:42.757993 1161 ts_tablet_manager.cc:1918] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.758132 1161 log.cc:1199] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9a90bb52358f4c38a540ad025e9e0430
I20250625 07:52:42.758435 1161 ts_tablet_manager.cc:1939] T 9a90bb52358f4c38a540ad025e9e0430 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.763103 1163 tablet_bootstrap.cc:492] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.763242 2407 ts_tablet_manager.cc:1918] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.763389 2407 log.cc:1199] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/486463a8140c4e769cc79870d2acff78
I20250625 07:52:42.763545 1163 ts_tablet_manager.cc:1397] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250625 07:52:42.763746 2407 ts_tablet_manager.cc:1939] T 486463a8140c4e769cc79870d2acff78 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:42.764534 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1496) took 2538 ms. Trace:
I20250625 07:52:42.766690 2723 raft_consensus.cc:491] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.766862 2723 raft_consensus.cc:513] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.767489 2723 leader_election.cc:290] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.768126 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c6d068d2324ea6a07bb9be74578870" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:42.768810 851 leader_election.cc:343] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.769388 1163 raft_consensus.cc:357] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.769476 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7c6d068d2324ea6a07bb9be74578870" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.769562 1163 raft_consensus.cc:383] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.769611 1163 raft_consensus.cc:738] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.769821 1163 consensus_queue.cc:260] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:42.770154 850 leader_election.cc:343] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.770249 850 leader_election.cc:304] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.770457 1163 ts_tablet_manager.cc:1428] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250625 07:52:42.770723 1163 tablet_bootstrap.cc:492] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.770995 2723 raft_consensus.cc:2747] T a7c6d068d2324ea6a07bb9be74578870 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.772367 1162 tablet_bootstrap.cc:492] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.772512 1163 tablet_bootstrap.cc:654] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.772558 1162 ts_tablet_manager.cc:1397] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250625 07:52:42.773391 1162 raft_consensus.cc:357] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.773752 2603 raft_consensus.cc:491] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.773888 2603 raft_consensus.cc:513] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.774679 2603 leader_election.cc:290] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.775313 1160 ts_tablet_manager.cc:1918] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.775554 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.775658 1162 raft_consensus.cc:383] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.775756 1162 raft_consensus.cc:738] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:42.776124 713 leader_election.cc:343] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.776445 1162 consensus_queue.cc:260] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.776641 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9a90bb52358f4c38a540ad025e9e0430 (table default.Table_b047e3c49b234a66991bed76ac01e6c6 [id=bead1aac10b84b00858d5a8768679257]) successfully deleted
I20250625 07:52:42.776988 1162 ts_tablet_manager.cc:1428] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250625 07:52:42.777835 1162 tablet_bootstrap.cc:492] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.764659 2407 rpcz_store.cc:276] 0625 07:52:40.226078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.226140 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:42.707763 (+2481623us) ts_tablet_manager.cc:1145] Deleting tablet 486463a8140c4e769cc79870d2acff78
0625 07:52:42.707773 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.745854 (+ 38081us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.763079 (+ 17225us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.764485 (+ 1406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481392}
I20250625 07:52:42.778991 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 486463a8140c4e769cc79870d2acff78 (table default.Table_5e910ce70a934be2be117a115fea3380 [id=1ecc6b62c7c140f1bff20d234b427666]) successfully deleted
W20250625 07:52:42.779120 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b85adedf7d2c45a1b33ee4a5633ee734 consensus state has no leader
I20250625 07:52:42.779484 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_42abf7d224a54cadb847ae4dd5d5c2a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.780054 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42abf7d224a54cadb847ae4dd5d5c2a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.780179 1162 tablet_bootstrap.cc:654] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.780901 1158 tablet_bootstrap.cc:492] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.781047 1158 ts_tablet_manager.cc:1397] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250625 07:52:42.781651 1158 raft_consensus.cc:357] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.781817 1158 raft_consensus.cc:383] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.781862 1158 raft_consensus.cc:738] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.780175 2407 tablet_replica.cc:331] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:42.782047 1158 consensus_queue.cc:260] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.782253 2407 raft_consensus.cc:2241] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.782632 1158 ts_tablet_manager.cc:1428] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250625 07:52:42.788115 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:52:42.788911 2836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:42.775722 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:42.777025 2723 raft_consensus.cc:491] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.789798 909 raft_consensus.cc:2466] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
I20250625 07:52:42.789804 2723 raft_consensus.cc:513] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.790354 713 leader_election.cc:304] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.790467 2723 leader_election.cc:290] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:42.790808 2603 raft_consensus.cc:2802] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:42.790896 2603 raft_consensus.cc:491] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:42.790954 2603 raft_consensus.cc:3058] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.791178 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec1e58fe6d72451682bd47d7238560b6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:42.791594 1160 log.cc:1199] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/51c584e6bffb4f0a8020dadf78d718a7
W20250625 07:52:42.791671 851 leader_election.cc:343] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.791960 1160 ts_tablet_manager.cc:1939] T 51c584e6bffb4f0a8020dadf78d718a7 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.793697 2603 raft_consensus.cc:513] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.794376 2603 leader_election.cc:290] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.800913 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:42.801265 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec1e58fe6d72451682bd47d7238560b6" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.801430 1158 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.802088 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62aaff3c000e4e69b52222aa47260c3a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:42.802256 914 raft_consensus.cc:3058] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
W20250625 07:52:42.802486 713 leader_election.cc:343] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.802690 713 consensus_peers.cc:487] T d6f04229a1d8462c818d589f9f4eb712 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:42.805436 850 leader_election.cc:343] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.805562 850 leader_election.cc:304] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.806351 2723 raft_consensus.cc:2747] T ec1e58fe6d72451682bd47d7238560b6 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.806980 914 raft_consensus.cc:2466] T 62aaff3c000e4e69b52222aa47260c3a P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:42.806074 591 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d45cd26e5aab475199c9c4b5ffb4d900" } new_table_name: "default.Table_4e64720a29e54d8e82de8b34e953edd7" modify_external_catalogs: true
I20250625 07:52:42.807406 713 leader_election.cc:304] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:42.807439 2835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:52:42.807837 2718 raft_consensus.cc:2802] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:42.807955 2718 raft_consensus.cc:695] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:42.808184 2718 consensus_queue.cc:237] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.808563 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1429) took 2629 ms. Trace:
I20250625 07:52:42.808640 1160 rpcz_store.cc:276] 0625 07:52:40.178707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.178761 (+ 54us) service_pool.cc:225] Handling call
0625 07:52:42.724771 (+2546010us) ts_tablet_manager.cc:1145] Deleting tablet 51c584e6bffb4f0a8020dadf78d718a7
0625 07:52:42.724781 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.752679 (+ 27898us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.775202 (+ 22523us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.808544 (+ 33342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2545820}
I20250625 07:52:42.810057 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 51c584e6bffb4f0a8020dadf78d718a7 (table default.Table_f3cd40e7670e492bba639148deda5ffd [id=493afd31fad94b3586421e8f7fbde02d]) successfully deleted
I20250625 07:52:42.813293 1161 tablet_replica.cc:331] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.813483 1161 raft_consensus.cc:2241] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.813622 1161 raft_consensus.cc:2270] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.814150 1161 ts_tablet_manager.cc:1905] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.817466 1160 tablet_replica.cc:331] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.819090 1160 raft_consensus.cc:2241] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250625 07:52:42.822213 591 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 868106f037d2484e94eab2f0d6394f92 consensus state has no leader
I20250625 07:52:42.822616 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_1bc954283dda4166b138c49523ff1417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.823247 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc954283dda4166b138c49523ff1417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.821596 1163 tablet_bootstrap.cc:492] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:42.824931 1160 raft_consensus.cc:2270] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.826897 1158 tablet_bootstrap.cc:654] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.825004 1163 ts_tablet_manager.cc:1397] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.003s
I20250625 07:52:42.827844 1163 raft_consensus.cc:357] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.828305 1163 raft_consensus.cc:383] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.828430 1163 raft_consensus.cc:738] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.828761 1163 consensus_queue.cc:260] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.832481 1163 ts_tablet_manager.cc:1428] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250625 07:52:42.832870 1163 tablet_bootstrap.cc:492] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:42.834043 2662 server_base.cc:1048] running on GCE node
I20250625 07:52:42.834818 1163 tablet_bootstrap.cc:654] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.835458 2662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:52:42.838397 1162 tablet_bootstrap.cc:492] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.838539 1162 ts_tablet_manager.cc:1397] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.005s
I20250625 07:52:42.839540 1162 raft_consensus.cc:357] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.839718 1162 raft_consensus.cc:383] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.839782 1162 raft_consensus.cc:738] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.840051 1162 consensus_queue.cc:260] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.835426 1160 ts_tablet_manager.cc:1905] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.841396 1162 ts_tablet_manager.cc:1428] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:42.841899 591 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c8893ee33bd4f88ac5c913e99fb4262 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:42.842404 1162 tablet_bootstrap.cc:492] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.843109 2612 raft_consensus.cc:491] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.843225 2612 raft_consensus.cc:513] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.843739 2612 leader_election.cc:290] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.844656 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22411725aa5741c497328ce047c44d50" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.845296 713 leader_election.cc:343] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 22411725aa5741c497328ce047c44d50
W20250625 07:52:42.845506 713 leader_election.cc:343] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.845606 713 leader_election.cc:304] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.846345 590 catalog_manager.cc:5582] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:42.846936 590 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 62aaff3c000e4e69b52222aa47260c3a (table default.Table_e01d3c58925248249dc9bff2b7be111c [id=79d68081c3a643109237f5d5478df82c]). Expected version 1 got 0
I20250625 07:52:42.847456 1161 ts_tablet_manager.cc:1918] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.847554 1161 log.cc:1199] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9f1d251171314118b66e3a921a4a50c5
I20250625 07:52:42.847980 2612 raft_consensus.cc:2747] T 22411725aa5741c497328ce047c44d50 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.850203 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 803792fd17a448c6b381bf2686f743a3 (DEFAULT_TABLE table=default.Table_6158c39c210841bc97e6b5a7c6e0b408 [id=b7b343f4b6124ba0927a69e2d4d20faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.850201 740 tablet_service.cc:1468] Processing CreateTablet for tablet 803792fd17a448c6b381bf2686f743a3 (DEFAULT_TABLE table=default.Table_6158c39c210841bc97e6b5a7c6e0b408 [id=b7b343f4b6124ba0927a69e2d4d20faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.850574 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803792fd17a448c6b381bf2686f743a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.850574 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803792fd17a448c6b381bf2686f743a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.851621 1162 tablet_bootstrap.cc:654] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.859206 1162 tablet_bootstrap.cc:492] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.859354 1162 ts_tablet_manager.cc:1397] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250625 07:52:42.860102 1162 raft_consensus.cc:357] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.860209 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 3f52c6475a644519828e7947e3dd46b1
I20250625 07:52:42.860255 1162 raft_consensus.cc:383] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.860306 1162 raft_consensus.cc:738] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.860930 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_342d570322ae4deb9b27e789a33a7493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.861449 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_342d570322ae4deb9b27e789a33a7493 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.861495 596 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4718c51c79ac41cebfca7dcc567ddecd
I20250625 07:52:42.862084 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_1463633d3ae24de887c29fc90b7c4c40" } new_table_name: "default.Table_95779e115782483593e578d791e7e3c7" modify_external_catalogs: true
I20250625 07:52:42.862563 593 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 4b2e211834bf4a1591759658e0f8e2be
I20250625 07:52:42.863096 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_064987a96c8f461f9f7a2e5430d0bca3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.863226 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 535259cb581843568b1561c607a7d542
W20250625 07:52:42.863549 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_064987a96c8f461f9f7a2e5430d0bca3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.863771 595 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3aebb6670cd64a50af22d28be0a65e47" } new_table_name: "default.Table_3a19ed2497be4604b30a16f8780b2d13" modify_external_catalogs: true
I20250625 07:52:42.865373 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f52c6475a644519828e7947e3dd46b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.865574 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.865659 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 535259cb581843568b1561c607a7d542 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.865696 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b2e211834bf4a1591759658e0f8e2be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.865947 1162 consensus_queue.cc:260] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.866758 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b2e211834bf4a1591759658e0f8e2be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.866837 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.866979 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 535259cb581843568b1561c607a7d542 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.867295 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f52c6475a644519828e7947e3dd46b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.867496 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4718c51c79ac41cebfca7dcc567ddecd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.867655 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b2e211834bf4a1591759658e0f8e2be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.867697 1162 ts_tablet_manager.cc:1428] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250625 07:52:42.867796 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 535259cb581843568b1561c607a7d542 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.867939 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f52c6475a644519828e7947e3dd46b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.868793 1162 tablet_bootstrap.cc:492] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.873018 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet a8f25febb41c403ca8be268206157756
I20250625 07:52:42.873337 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fe3ccad4a5a34bfba36c5c18ec5038c0
I20250625 07:52:42.873407 589 catalog_manager.cc:6652] ReplaceTablet: tablet a8f25febb41c403ca8be268206157756 deleted and replaced by tablet 4864c3b5c30f4ad0bd6bc33232c6d752
I20250625 07:52:42.873747 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_85ad0c3916d74cf698781b7ee0d3017d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:42.874104 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0bfa1c4d4b6e4530892118189a5ab516 (table default.Table_6b88554443774ac1816a1e90b766e23d [id=e10e5f4032ce4e51b479b74f9193de5f]). Expected version 1 got 0
I20250625 07:52:42.874056 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2e6ed1dba6e54a57b1be21a0fb016f66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.874215 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85ad0c3916d74cf698781b7ee0d3017d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.874282 597 catalog_manager.cc:5582] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 429026e8d98d46339db143f84897a092 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "429026e8d98d46339db143f84897a092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: HEALTHY } } }
W20250625 07:52:42.874492 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e6ed1dba6e54a57b1be21a0fb016f66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.874596 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
I20250625 07:52:42.874976 597 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 15766b87fb1b4620970d5fab21465cc4 (table default.Table_905ed787fc744bffb016385b3eb25732 [id=d4be884fab4f4468a41fb5a7eff91ad0]). Expected version 2 got 1
I20250625 07:52:42.875490 594 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_f8984bdccae14f52a39c0e273f526966 [id=b0f32fb69f33447db7620299affa88c8] alter complete (version 4)
I20250625 07:52:42.876539 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a8f25febb41c403ca8be268206157756 with delete_type TABLET_DATA_DELETED (replaced by tablet 4864c3b5c30f4ad0bd6bc33232c6d752) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.876520 740 tablet_service.cc:1515] Processing DeleteTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.876789 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_dfa55dcd1e7047b3b00012f47e6c897b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.877228 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfa55dcd1e7047b3b00012f47e6c897b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.877482 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bccda893fd5466886a4d253172309e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.877482 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.877769 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a8f25febb41c403ca8be268206157756 with delete_type TABLET_DATA_DELETED (replaced by tablet 4864c3b5c30f4ad0bd6bc33232c6d752) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.877960 887 tablet_service.cc:1515] Processing DeleteTablet for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:42 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.878232 591 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 6c8893ee33bd4f88ac5c913e99fb4262
I20250625 07:52:42.881435 591 catalog_manager.cc:6652] ReplaceTablet: tablet 6c8893ee33bd4f88ac5c913e99fb4262 deleted and replaced by tablet d0e082d6bc974923b2aad04a837bbb44
I20250625 07:52:42.881755 1162 tablet_bootstrap.cc:654] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.881803 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e082d6bc974923b2aad04a837bbb44) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:42.881951 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e082d6bc974923b2aad04a837bbb44) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.882033 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c8893ee33bd4f88ac5c913e99fb4262 with delete_type TABLET_DATA_DELETED (replaced by tablet d0e082d6bc974923b2aad04a837bbb44) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.882897 1160 ts_tablet_manager.cc:1918] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.883503 1160 log.cc:1199] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b5bc347c278c44c69444caf9ded72e1a
I20250625 07:52:42.883967 1160 ts_tablet_manager.cc:1939] T b5bc347c278c44c69444caf9ded72e1a P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:42.884130 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet a8f25febb41c403ca8be268206157756 with delete_type TABLET_DATA_DELETED (replaced by tablet 4864c3b5c30f4ad0bd6bc33232c6d752) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:42.885480 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1430) took 2706 ms. Trace:
I20250625 07:52:42.886353 1160 rpcz_store.cc:276] 0625 07:52:40.179264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.179313 (+ 49us) service_pool.cc:225] Handling call
0625 07:52:42.817409 (+2638096us) ts_tablet_manager.cc:1145] Deleting tablet b5bc347c278c44c69444caf9ded72e1a
0625 07:52:42.817420 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.849586 (+ 32166us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.882771 (+ 33185us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:42.885459 (+ 2688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2637890}
I20250625 07:52:42.887107 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet b5bc347c278c44c69444caf9ded72e1a (table default.Table_6a60665ba1c74648904cfad10762f2c0 [id=a4bf0e1d16dd4f529cfe1a204c67123a]) successfully deleted
I20250625 07:52:42.887526 1160 tablet_replica.cc:331] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:42.890062 1160 raft_consensus.cc:2241] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.891048 1158 tablet_bootstrap.cc:492] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:42.891181 1158 ts_tablet_manager.cc:1397] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.003s
I20250625 07:52:42.891454 1160 raft_consensus.cc:2270] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.891801 1158 raft_consensus.cc:357] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.891973 1158 raft_consensus.cc:383] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.892026 1158 raft_consensus.cc:738] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.892213 1158 consensus_queue.cc:260] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.892652 1158 ts_tablet_manager.cc:1428] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250625 07:52:42.892971 1158 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:42.894618 1158 tablet_bootstrap.cc:654] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.894939 1160 ts_tablet_manager.cc:1905] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.896571 1161 ts_tablet_manager.cc:1939] T 9f1d251171314118b66e3a921a4a50c5 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.897022 1161 tablet_replica.cc:331] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.897253 1161 raft_consensus.cc:2241] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.897426 1161 raft_consensus.cc:2270] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.901401 1161 ts_tablet_manager.cc:1905] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.901857 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9f1d251171314118b66e3a921a4a50c5 (table default.Table_1437af2855fc49d6b981d94261019399 [id=4eec70fdea6f418fa045a2d926f2eca6]) successfully deleted
I20250625 07:52:42.905232 2662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:52:42.907094 2718 raft_consensus.cc:491] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.907254 2718 raft_consensus.cc:513] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.907846 2718 leader_election.cc:290] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.908591 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "484a49fab6134d059c64f5a3f2b29e03" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:42.909139 713 leader_election.cc:343] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:42.909968 1163 tablet_bootstrap.cc:492] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
W20250625 07:52:42.910036 713 leader_election.cc:343] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 484a49fab6134d059c64f5a3f2b29e03
I20250625 07:52:42.910143 713 leader_election.cc:304] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.910292 1163 ts_tablet_manager.cc:1397] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.000s
I20250625 07:52:42.910353 2718 raft_consensus.cc:2747] T 484a49fab6134d059c64f5a3f2b29e03 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.911504 1163 raft_consensus.cc:357] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.911728 1163 raft_consensus.cc:383] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.911823 1163 raft_consensus.cc:738] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.912092 1163 consensus_queue.cc:260] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.915778 1163 ts_tablet_manager.cc:1428] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250625 07:52:42.916147 1163 tablet_bootstrap.cc:492] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:42.918028 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53b34a2db6a345a6a22b28cd0435d95f consensus state has no leader
I20250625 07:52:42.918087 1162 tablet_bootstrap.cc:492] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.918222 1162 ts_tablet_manager.cc:1397] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.001s
I20250625 07:52:42.918458 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_573c13bcee8d48e285016445b965e0ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.918979 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_573c13bcee8d48e285016445b965e0ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.918987 1162 raft_consensus.cc:357] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.919159 1162 raft_consensus.cc:383] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.919212 1162 raft_consensus.cc:738] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.919406 1162 consensus_queue.cc:260] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.919770 592 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_cc3b3771179f4d57b2d2a7e965e87550" } new_table_name: "default.Table_f71ab2d8b8804cc38089620ad86c746b" modify_external_catalogs: true
I20250625 07:52:42.920028 1162 ts_tablet_manager.cc:1428] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:42.920293 1162 tablet_bootstrap.cc:492] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.925668 1162 tablet_bootstrap.cc:654] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.927502 1163 tablet_bootstrap.cc:654] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:42.933424 1161 ts_tablet_manager.cc:1918] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:42.933568 1161 log.cc:1199] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/3823279d7e06435484f1b82836ebcf39
I20250625 07:52:42.933845 1161 ts_tablet_manager.cc:1939] T 3823279d7e06435484f1b82836ebcf39 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:42.934360 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 3823279d7e06435484f1b82836ebcf39 (table default.Table_69c013cd0c6a4f5397f00282c3269577 [id=9197109c61c648aeb670a20ef196bb30]) successfully deleted
I20250625 07:52:42.937659 2662 hybrid_clock.cc:648] HybridClock initialized: now 1750837962937582 us; error 57 us; skew 500 ppm
I20250625 07:52:42.938206 2662 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:52:42.939469 887 tablet_service.cc:1468] Processing CreateTablet for tablet 803792fd17a448c6b381bf2686f743a3 (DEFAULT_TABLE table=default.Table_6158c39c210841bc97e6b5a7c6e0b408 [id=b7b343f4b6124ba0927a69e2d4d20faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.939901 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803792fd17a448c6b381bf2686f743a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.940021 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7a753c565b7349558d3722c8376208c7 (DEFAULT_TABLE table=default.Table_abf60fb31fbd42b5ad3b47e078e00649 [id=2e5b1e303b4442c0826607350c196d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.940371 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a753c565b7349558d3722c8376208c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.941438 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 8793c5237be044749eb93e2b92f8500e (DEFAULT_TABLE table=default.Table_106241d2c1a44caca30ba7a0a604059f [id=21984e04427e4fc7910b55ad55530959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.941753 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8793c5237be044749eb93e2b92f8500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.945122 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7a753c565b7349558d3722c8376208c7 (DEFAULT_TABLE table=default.Table_abf60fb31fbd42b5ad3b47e078e00649 [id=2e5b1e303b4442c0826607350c196d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.945538 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a753c565b7349558d3722c8376208c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.947558 2718 raft_consensus.cc:491] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:42.947690 2718 raft_consensus.cc:513] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.948313 2718 leader_election.cc:290] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:42.948786 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b04a575e0984f879f0179c8baa3f3fa" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:42.948993 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7a753c565b7349558d3722c8376208c7 (DEFAULT_TABLE table=default.Table_abf60fb31fbd42b5ad3b47e078e00649 [id=2e5b1e303b4442c0826607350c196d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.949362 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a753c565b7349558d3722c8376208c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:42.949564 713 leader_election.cc:343] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:42.949894 713 leader_election.cc:343] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 5b04a575e0984f879f0179c8baa3f3fa
I20250625 07:52:42.949982 713 leader_election.cc:304] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:42.950194 2718 raft_consensus.cc:2747] T 5b04a575e0984f879f0179c8baa3f3fa P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:42.953802 1161 tablet_replica.cc:331] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:42.953994 1161 raft_consensus.cc:2241] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:42.954183 1161 raft_consensus.cc:2270] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.954605 1047 raft_consensus.cc:3058] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:42.954608 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8793c5237be044749eb93e2b92f8500e (DEFAULT_TABLE table=default.Table_106241d2c1a44caca30ba7a0a604059f [id=21984e04427e4fc7910b55ad55530959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.954986 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8793c5237be044749eb93e2b92f8500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.963816 2407 raft_consensus.cc:2270] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:42.964560 2407 ts_tablet_manager.cc:1905] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.965466 894 tablet_service.cc:1468] Processing CreateTablet for tablet 8793c5237be044749eb93e2b92f8500e (DEFAULT_TABLE table=default.Table_106241d2c1a44caca30ba7a0a604059f [id=21984e04427e4fc7910b55ad55530959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.965824 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 38c221d71bff4b949596b36cb3a1ab80 (DEFAULT_TABLE table=default.Table_e703a594772745af9cc4330052da8807 [id=02dcb595107d4a209fa72d28d77c6f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.965929 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8793c5237be044749eb93e2b92f8500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.968765 1161 ts_tablet_manager.cc:1905] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:42.970060 752 tablet_service.cc:1468] Processing CreateTablet for tablet 38c221d71bff4b949596b36cb3a1ab80 (DEFAULT_TABLE table=default.Table_e703a594772745af9cc4330052da8807 [id=02dcb595107d4a209fa72d28d77c6f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.970496 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c221d71bff4b949596b36cb3a1ab80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.969259 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c221d71bff4b949596b36cb3a1ab80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.975611 887 tablet_service.cc:1468] Processing CreateTablet for tablet 38c221d71bff4b949596b36cb3a1ab80 (DEFAULT_TABLE table=default.Table_e703a594772745af9cc4330052da8807 [id=02dcb595107d4a209fa72d28d77c6f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.976024 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38c221d71bff4b949596b36cb3a1ab80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.976789 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 67f8ae5b56ef49e594befd1caec5b049 (DEFAULT_TABLE table=default.Table_12ba62aefbe74bafa70d60756d1e0b0d [id=b7a3cce7018f4aa9aa6c4b98c0415c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:42.977540 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f8ae5b56ef49e594befd1caec5b049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:42.981287 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_f3af36fb7fbd495487db43d049d58821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.981818 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3af36fb7fbd495487db43d049d58821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:42.983065 595 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1463d047d40a49688267e9e396356841 consensus state has no leader
I20250625 07:52:42.983368 595 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_64df716988694120a8b3194dba2d025f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:42.983812 595 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64df716988694120a8b3194dba2d025f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:42.985919 1162 tablet_bootstrap.cc:492] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:42.986099 1162 ts_tablet_manager.cc:1397] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20250625 07:52:42.986838 1162 raft_consensus.cc:357] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.987027 1162 raft_consensus.cc:383] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:42.987080 1162 raft_consensus.cc:738] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:42.987291 1162 consensus_queue.cc:260] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:42.987831 1162 ts_tablet_manager.cc:1428] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:42.988437 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:42.988653 1162 tablet_bootstrap.cc:492] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:42.989957 590 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_905ed787fc744bffb016385b3eb25732 [id=d4be884fab4f4468a41fb5a7eff91ad0] alter complete (version 2)
I20250625 07:52:42.991302 1162 tablet_bootstrap.cc:654] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:42.994791 592 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f37ef18e9124e50bcfecf9631cef7ea consensus state has no leader
I20250625 07:52:42.995401 588 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9a56722f526b4aceb8716901c46a8be5" } new_table_name: "default.Table_32f6e693d0304ee8a0c3041edf1b7bfc" modify_external_catalogs: true
I20250625 07:52:42.995648 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 17be34eba92246df973f3d464e13fd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.996536 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 22411725aa5741c497328ce047c44d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.996728 754 tablet_service.cc:1515] Processing DeleteTablet for tablet e4b87309b76f4f0e9dde120d9f755c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:39 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:42.999889 2612 raft_consensus.cc:491] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.000022 2612 raft_consensus.cc:513] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.000312 1160 ts_tablet_manager.cc:1918] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.000424 1160 log.cc:1199] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/9578d77db00242d48f4e8f6f8263ef09
I20250625 07:52:43.000589 2612 leader_election.cc:290] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.000767 1160 ts_tablet_manager.cc:1939] T 9578d77db00242d48f4e8f6f8263ef09 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:43.001575 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 962633b0d9644c308f0b7fe1db714daf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:43.003271 713 leader_election.cc:343] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: e06ecad6c7464ffe8ec6f632634307c9
I20250625 07:52:43.004208 1046 raft_consensus.cc:3058] T ffd98b2877b944f0bd484efa2a38a00a P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:43.005929 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1431) took 2801 ms. Trace:
I20250625 07:52:43.006021 1160 rpcz_store.cc:276] 0625 07:52:40.204505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.204566 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:42.887473 (+2682907us) ts_tablet_manager.cc:1145] Deleting tablet 9578d77db00242d48f4e8f6f8263ef09
0625 07:52:42.887484 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.930650 (+ 43166us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.000182 (+ 69532us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.005906 (+ 5724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2682684}
I20250625 07:52:43.006386 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 9578d77db00242d48f4e8f6f8263ef09 (table default.Table_29b24dec640147299371e6748d35cc5b [id=6041ac7669374b268f40cadc19984959]) successfully deleted
I20250625 07:52:43.007350 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3758b4fc14df40f3bb9a6e77699981ea with delete_type TABLET_DATA_DELETED (replaced by tablet 387a69222770408598184411225340e7) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.007604 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e06ecad6c7464ffe8ec6f632634307c9" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.008136 713 leader_election.cc:343] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.008258 713 leader_election.cc:304] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.009076 2612 raft_consensus.cc:2747] T e06ecad6c7464ffe8ec6f632634307c9 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.012300 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_bfe4dd4209ea4ace8bd9414cab6da3d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.012940 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfe4dd4209ea4ace8bd9414cab6da3d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.017318 754 tablet_service.cc:1468] Processing CreateTablet for tablet 67f8ae5b56ef49e594befd1caec5b049 (DEFAULT_TABLE table=default.Table_12ba62aefbe74bafa70d60756d1e0b0d [id=b7a3cce7018f4aa9aa6c4b98c0415c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.017858 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f8ae5b56ef49e594befd1caec5b049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.025565 2407 ts_tablet_manager.cc:1918] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.027418 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 896b9a3fe00a4dc491cb4de0d5201c1b (DEFAULT_TABLE table=default.Table_89bf4acb52434e51a773e4ff8b98b203 [id=c65c67128a30454fbdc4ab4b356e4119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.027439 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb2fba569e445d498780d862197d3aa (DEFAULT_TABLE table=default.Table_0e7144236c034538bb54d0c2d84c9b4b [id=beb74ba52f9244b7b9eec9c7420fe749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.027890 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896b9a3fe00a4dc491cb4de0d5201c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.028285 884 tablet_service.cc:1468] Processing CreateTablet for tablet 69a9c299a4bf4567a6dfaaaf346556c4 (DEFAULT_TABLE table=default.Table_66c399c228be41b2888b89548b2d5738 [id=bb63a6f48fe74b1c9ee7e17b3d44910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.025899 740 tablet_service.cc:1468] Processing CreateTablet for tablet 896b9a3fe00a4dc491cb4de0d5201c1b (DEFAULT_TABLE table=default.Table_89bf4acb52434e51a773e4ff8b98b203 [id=c65c67128a30454fbdc4ab4b356e4119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.026082 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb2fba569e445d498780d862197d3aa (DEFAULT_TABLE table=default.Table_0e7144236c034538bb54d0c2d84c9b4b [id=beb74ba52f9244b7b9eec9c7420fe749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.028463 891 tablet_service.cc:1468] Processing CreateTablet for tablet 896b9a3fe00a4dc491cb4de0d5201c1b (DEFAULT_TABLE table=default.Table_89bf4acb52434e51a773e4ff8b98b203 [id=c65c67128a30454fbdc4ab4b356e4119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.028286 878 tablet_service.cc:1468] Processing CreateTablet for tablet 67f8ae5b56ef49e594befd1caec5b049 (DEFAULT_TABLE table=default.Table_12ba62aefbe74bafa70d60756d1e0b0d [id=b7a3cce7018f4aa9aa6c4b98c0415c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.025900 739 tablet_service.cc:1468] Processing CreateTablet for tablet 69a9c299a4bf4567a6dfaaaf346556c4 (DEFAULT_TABLE table=default.Table_66c399c228be41b2888b89548b2d5738 [id=bb63a6f48fe74b1c9ee7e17b3d44910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.028846 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896b9a3fe00a4dc491cb4de0d5201c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.029006 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2fba569e445d498780d862197d3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.031847 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c299a4bf4567a6dfaaaf346556c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.033200 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2fba569e445d498780d862197d3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.036931 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 69a9c299a4bf4567a6dfaaaf346556c4 (DEFAULT_TABLE table=default.Table_66c399c228be41b2888b89548b2d5738 [id=bb63a6f48fe74b1c9ee7e17b3d44910b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.037372 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c299a4bf4567a6dfaaaf346556c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.039618 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5a8a06415cdf4601a5b6749ad29a2a19" } new_table_name: "default.Table_d83a39ca1c554b46ad5d6a27d4d89226" modify_external_catalogs: true
I20250625 07:52:43.041494 1163 tablet_bootstrap.cc:492] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:43.041702 1163 ts_tablet_manager.cc:1397] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.004s
I20250625 07:52:43.042022 592 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_ac4d9f201936430cb7c72ed39467bf3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.042529 592 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4d9f201936430cb7c72ed39467bf3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.042529 1163 raft_consensus.cc:357] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.042678 1163 raft_consensus.cc:383] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.042738 1163 raft_consensus.cc:738] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.042982 1163 consensus_queue.cc:260] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.044130 1163 ts_tablet_manager.cc:1428] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250625 07:52:43.044333 2407 log.cc:1199] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/44bfcc2b20ba4d0b83641988f2c0ee35
I20250625 07:52:43.044785 2407 ts_tablet_manager.cc:1939] T 44bfcc2b20ba4d0b83641988f2c0ee35 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:43.045377 595 master_service.cc:948] ReplaceTablet: received request to replace tablet 44906d1dd7a84ec8bcc6701bdd2a1872 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:43.045436 883 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb2fba569e445d498780d862197d3aa (DEFAULT_TABLE table=default.Table_0e7144236c034538bb54d0c2d84c9b4b [id=beb74ba52f9244b7b9eec9c7420fe749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.046165 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1497) took 2819 ms. Trace:
I20250625 07:52:43.046351 2407 rpcz_store.cc:276] 0625 07:52:40.226212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.227457 (+ 1245us) service_pool.cc:225] Handling call
0625 07:52:42.780082 (+2552625us) ts_tablet_manager.cc:1145] Deleting tablet 44bfcc2b20ba4d0b83641988f2c0ee35
0625 07:52:42.780099 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:42.999527 (+219428us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.025453 (+ 25926us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.046143 (+ 20690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2550783}
I20250625 07:52:43.047995 2407 tablet_replica.cc:331] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.048236 2407 raft_consensus.cc:2241] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.048424 2407 raft_consensus.cc:2270] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.049046 2407 ts_tablet_manager.cc:1905] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.051987 596 catalog_manager.cc:5582] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:43.060107 596 catalog_manager.cc:5582] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:43.060432 1158 tablet_bootstrap.cc:492] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.060988 1158 ts_tablet_manager.cc:1397] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.168s user 0.003s sys 0.001s
I20250625 07:52:43.061367 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 62aaff3c000e4e69b52222aa47260c3a (table default.Table_e01d3c58925248249dc9bff2b7be111c [id=79d68081c3a643109237f5d5478df82c]). Expected version 1 got 0
I20250625 07:52:43.062433 1158 raft_consensus.cc:357] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.062619 1158 raft_consensus.cc:383] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.062680 1158 raft_consensus.cc:738] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.062960 1158 consensus_queue.cc:260] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.063642 1158 ts_tablet_manager.cc:1428] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250625 07:52:43.063915 1158 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.067890 596 catalog_manager.cc:5582] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:43.068744 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c175adca5b37414f9da37047d232e973 (DEFAULT_TABLE table=default.Table_f90edcd94960433cb8e76ccabc3e4e59 [id=103516739a04408d9d09ccdaa9f40153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.069162 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c175adca5b37414f9da37047d232e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.069718 596 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet d1c96cdddd3443b8b67f57f4d6c4846e (table default.Table_802e0f6519e044c0a5b420289568590f [id=1082e4c0af594912bd6b02bd57557cf0]). Expected version 1 got 0
I20250625 07:52:43.070410 596 catalog_manager.cc:5582] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:43.072126 2723 raft_consensus.cc:491] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.072274 2723 raft_consensus.cc:513] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.073091 2723 leader_election.cc:290] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.074106 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "962633b0d9644c308f0b7fe1db714daf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:43.074757 851 leader_election.cc:343] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.075356 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 2cb842878d8c48158a0729047d6b7da2 (DEFAULT_TABLE table=default.Table_d8ffd5ca03a642c699002e8361098a0a [id=b980add8c7b042318afd2417cdc46bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.075459 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3e2851494cfc44b09f0f248b51a11858 (DEFAULT_TABLE table=default.Table_26f258ae9fd74416bba2b873db0695aa [id=dfac577d06d3453782a52e4a35d264a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.075759 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cb842878d8c48158a0729047d6b7da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.075953 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e66aff696c084e379578b9350e4ac23d (DEFAULT_TABLE table=default.Table_ba58cf7f3f5943dc8d14d2f29fdb1684 [id=5df86cea157e4b40856bee5023864502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.076108 2799 tablet.cc:1722] T 896419bd407f43e1a0a361330ff9821b P 8f2cdaf87b9d4b2a919098bf65efd513: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250625 07:52:43.076267 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66aff696c084e379578b9350e4ac23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.076323 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 1da59ad9c6f14064b71581e08cacf7fc (DEFAULT_TABLE table=default.Table_b1925e38730243d2bacf4c3b8ae37393 [id=36d1470cb7364acb961459d3dc62f5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.076627 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da59ad9c6f14064b71581e08cacf7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.076939 1024 tablet_service.cc:1468] Processing CreateTablet for tablet eaa8b405985e4331bbdfa5c6d5f6518c (DEFAULT_TABLE table=default.Table_00a5bff9b246451ca66616d3b60fed60 [id=b6641dd9c8db4b7cacb3e777ace7c923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.077227 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa8b405985e4331bbdfa5c6d5f6518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.080957 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 2db5d25775ec48da8511146ac7df4068 (DEFAULT_TABLE table=default.Table_50f0c18132ce48cf8b4242ae079512e0 [id=57f8b5560c624e10bebb4549f1585793]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.081133 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9759b06b076d4d3aba740791256e8991 (DEFAULT_TABLE table=default.Table_cba2a3adedc14eb1a572c3f07e0709c0 [id=4414a0ed17814d1991b65a192c5dee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.081277 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db5d25775ec48da8511146ac7df4068. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.081422 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9759b06b076d4d3aba740791256e8991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.081694 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5d0f34ff108d480d87b95ffa135df86d (DEFAULT_TABLE table=default.Table_19f5f9aa6d8545839a1b66329c73df04 [id=20e7a70c0329435780b8b81681f861c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.081977 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0f34ff108d480d87b95ffa135df86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.084913 1021 tablet_service.cc:1468] Processing CreateTablet for tablet da1f526d6e3c48a58b34cb1ef0753876 (DEFAULT_TABLE table=default.Table_ea9b2b65ce2b451993172b74d520f24c [id=4e8850c115f144709db9ce1702f110e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.085107 1014 tablet_service.cc:1468] Processing CreateTablet for tablet ebd4b665b7b14aef856f53262ce7ae52 (DEFAULT_TABLE table=default.Table_9bda35c7fae042e7af6631324add64d8 [id=b760e637286c4ec39142b21a32d0e6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.085225 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da1f526d6e3c48a58b34cb1ef0753876. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.085399 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebd4b665b7b14aef856f53262ce7ae52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.085623 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 039992783bb147178880d0b77aeced0b (DEFAULT_TABLE table=default.Table_73ebfb29d78146feb6bb0b4ae5633ebc [id=da965d755e754eeea6a89643ca3245de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.085893 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039992783bb147178880d0b77aeced0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.089004 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2851494cfc44b09f0f248b51a11858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.084932 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 12db6e59e76f429f9be71ccc77c65d3a (DEFAULT_TABLE table=default.Table_f66d6700dcbf489286819ea4f80a75a3 [id=be93e5af63274dfe98559ebb8512479d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.090008 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db6e59e76f429f9be71ccc77c65d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.080953 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb (DEFAULT_TABLE table=default.Table_7afdcd9db0394301b8a6504a6adf9b0b [id=2f92f44fb69440aaa64033c42da81f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.093080 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.097162 2723 raft_consensus.cc:491] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.097318 2723 raft_consensus.cc:513] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.097622 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5d0f34ff108d480d87b95ffa135df86d (DEFAULT_TABLE table=default.Table_19f5f9aa6d8545839a1b66329c73df04 [id=20e7a70c0329435780b8b81681f861c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.097935 2723 leader_election.cc:290] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.098011 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0f34ff108d480d87b95ffa135df86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.098071 744 tablet_service.cc:1468] Processing CreateTablet for tablet eaa8b405985e4331bbdfa5c6d5f6518c (DEFAULT_TABLE table=default.Table_00a5bff9b246451ca66616d3b60fed60 [id=b6641dd9c8db4b7cacb3e777ace7c923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.098208 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9759b06b076d4d3aba740791256e8991 (DEFAULT_TABLE table=default.Table_cba2a3adedc14eb1a572c3f07e0709c0 [id=4414a0ed17814d1991b65a192c5dee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.098341 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa8b405985e4331bbdfa5c6d5f6518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.098376 750 tablet_service.cc:1468] Processing CreateTablet for tablet 2db5d25775ec48da8511146ac7df4068 (DEFAULT_TABLE table=default.Table_50f0c18132ce48cf8b4242ae079512e0 [id=57f8b5560c624e10bebb4549f1585793]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.098531 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9759b06b076d4d3aba740791256e8991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.097087 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1da59ad9c6f14064b71581e08cacf7fc (DEFAULT_TABLE table=default.Table_b1925e38730243d2bacf4c3b8ae37393 [id=36d1470cb7364acb961459d3dc62f5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.098903 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da59ad9c6f14064b71581e08cacf7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.100929 745 tablet_service.cc:1468] Processing CreateTablet for tablet da1f526d6e3c48a58b34cb1ef0753876 (DEFAULT_TABLE table=default.Table_ea9b2b65ce2b451993172b74d520f24c [id=4e8850c115f144709db9ce1702f110e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.101234 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da1f526d6e3c48a58b34cb1ef0753876. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.098208 748 tablet_service.cc:1468] Processing CreateTablet for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb (DEFAULT_TABLE table=default.Table_7afdcd9db0394301b8a6504a6adf9b0b [id=2f92f44fb69440aaa64033c42da81f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.104607 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.104786 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db5d25775ec48da8511146ac7df4068. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.104996 758 tablet_service.cc:1468] Processing CreateTablet for tablet 039992783bb147178880d0b77aeced0b (DEFAULT_TABLE table=default.Table_73ebfb29d78146feb6bb0b4ae5633ebc [id=da965d755e754eeea6a89643ca3245de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.105273 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039992783bb147178880d0b77aeced0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.097086 741 tablet_service.cc:1468] Processing CreateTablet for tablet e66aff696c084e379578b9350e4ac23d (DEFAULT_TABLE table=default.Table_ba58cf7f3f5943dc8d14d2f29fdb1684 [id=5df86cea157e4b40856bee5023864502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.108980 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66aff696c084e379578b9350e4ac23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.108953 757 tablet_service.cc:1468] Processing CreateTablet for tablet 12db6e59e76f429f9be71ccc77c65d3a (DEFAULT_TABLE table=default.Table_f66d6700dcbf489286819ea4f80a75a3 [id=be93e5af63274dfe98559ebb8512479d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.109246 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db6e59e76f429f9be71ccc77c65d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.111474 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_045259525d4b454fbc351c409b78e67a" } modify_external_catalogs: true
I20250625 07:52:43.111651 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_045259525d4b454fbc351c409b78e67a" } modify_external_catalogs: true
I20250625 07:52:43.112902 589 master_service.cc:948] ReplaceTablet: received request to replace tablet 07837660a7db47299b8e11f3335cb5d7 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:43.112962 756 tablet_service.cc:1468] Processing CreateTablet for tablet ebd4b665b7b14aef856f53262ce7ae52 (DEFAULT_TABLE table=default.Table_9bda35c7fae042e7af6631324add64d8 [id=b760e637286c4ec39142b21a32d0e6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.113258 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebd4b665b7b14aef856f53262ce7ae52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.115320 892 tablet_service.cc:1468] Processing CreateTablet for tablet c175adca5b37414f9da37047d232e973 (DEFAULT_TABLE table=default.Table_f90edcd94960433cb8e76ccabc3e4e59 [id=103516739a04408d9d09ccdaa9f40153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.115537 1012 tablet_service.cc:1468] Processing CreateTablet for tablet 254607ebd73c4f508597b5c6c474f445 (DEFAULT_TABLE table=default.Table_c194642288ef486a9d9aac5a144c385e [id=52729ba430b344c8903802c0a5abc5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.115705 886 tablet_service.cc:1468] Processing CreateTablet for tablet 2cb842878d8c48158a0729047d6b7da2 (DEFAULT_TABLE table=default.Table_d8ffd5ca03a642c699002e8361098a0a [id=b980add8c7b042318afd2417cdc46bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.115922 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254607ebd73c4f508597b5c6c474f445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.116067 876 tablet_service.cc:1468] Processing CreateTablet for tablet 7a78321046a444bcbe8dd7773b39d6c7 (DEFAULT_TABLE table=default.Table_7744264852c74b1383d4855dcc2f149e [id=1e891425c2c64d44bbefd09310adb93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.116416 881 tablet_service.cc:1468] Processing CreateTablet for tablet e66aff696c084e379578b9350e4ac23d (DEFAULT_TABLE table=default.Table_ba58cf7f3f5943dc8d14d2f29fdb1684 [id=5df86cea157e4b40856bee5023864502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.116906 739 tablet_service.cc:1468] Processing CreateTablet for tablet c175adca5b37414f9da37047d232e973 (DEFAULT_TABLE table=default.Table_f90edcd94960433cb8e76ccabc3e4e59 [id=103516739a04408d9d09ccdaa9f40153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.116930 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1da59ad9c6f14064b71581e08cacf7fc (DEFAULT_TABLE table=default.Table_b1925e38730243d2bacf4c3b8ae37393 [id=36d1470cb7364acb961459d3dc62f5ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.117048 885 tablet_service.cc:1468] Processing CreateTablet for tablet 2db5d25775ec48da8511146ac7df4068 (DEFAULT_TABLE table=default.Table_50f0c18132ce48cf8b4242ae079512e0 [id=57f8b5560c624e10bebb4549f1585793]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.117234 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c175adca5b37414f9da37047d232e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.117416 880 tablet_service.cc:1468] Processing CreateTablet for tablet 3e2851494cfc44b09f0f248b51a11858 (DEFAULT_TABLE table=default.Table_26f258ae9fd74416bba2b873db0695aa [id=dfac577d06d3453782a52e4a35d264a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.117820 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a5a40faa418420ea31bc1d46cab1f4a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.117920 890 tablet_service.cc:1468] Processing CreateTablet for tablet eaa8b405985e4331bbdfa5c6d5f6518c (DEFAULT_TABLE table=default.Table_00a5bff9b246451ca66616d3b60fed60 [id=b6641dd9c8db4b7cacb3e777ace7c923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.118511 882 tablet_service.cc:1468] Processing CreateTablet for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb (DEFAULT_TABLE table=default.Table_7afdcd9db0394301b8a6504a6adf9b0b [id=2f92f44fb69440aaa64033c42da81f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.118557 2612 raft_consensus.cc:491] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.118680 2612 raft_consensus.cc:513] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.118933 877 tablet_service.cc:1468] Processing CreateTablet for tablet 9759b06b076d4d3aba740791256e8991 (DEFAULT_TABLE table=default.Table_cba2a3adedc14eb1a572c3f07e0709c0 [id=4414a0ed17814d1991b65a192c5dee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.119238 2612 leader_election.cc:290] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.119458 895 tablet_service.cc:1468] Processing CreateTablet for tablet 5d0f34ff108d480d87b95ffa135df86d (DEFAULT_TABLE table=default.Table_19f5f9aa6d8545839a1b66329c73df04 [id=20e7a70c0329435780b8b81681f861c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.119537 879 tablet_service.cc:1468] Processing CreateTablet for tablet da1f526d6e3c48a58b34cb1ef0753876 (DEFAULT_TABLE table=default.Table_ea9b2b65ce2b451993172b74d520f24c [id=4e8850c115f144709db9ce1702f110e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.119813 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a5a40faa418420ea31bc1d46cab1f4a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.119992 893 tablet_service.cc:1468] Processing CreateTablet for tablet 039992783bb147178880d0b77aeced0b (DEFAULT_TABLE table=default.Table_73ebfb29d78146feb6bb0b4ae5633ebc [id=da965d755e754eeea6a89643ca3245de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.120117 888 tablet_service.cc:1468] Processing CreateTablet for tablet 12db6e59e76f429f9be71ccc77c65d3a (DEFAULT_TABLE table=default.Table_f66d6700dcbf489286819ea4f80a75a3 [id=be93e5af63274dfe98559ebb8512479d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:43.120311 851 leader_election.cc:343] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:43.120528 850 leader_election.cc:343] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.120625 850 leader_election.cc:304] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.120976 2723 raft_consensus.cc:491] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.121412 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3e2851494cfc44b09f0f248b51a11858 (DEFAULT_TABLE table=default.Table_26f258ae9fd74416bba2b873db0695aa [id=dfac577d06d3453782a52e4a35d264a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.121742 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2851494cfc44b09f0f248b51a11858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.121953 2723 raft_consensus.cc:513] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.122509 2723 leader_election.cc:290] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.122637 2723 raft_consensus.cc:2747] T 1a5a40faa418420ea31bc1d46cab1f4a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.123207 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d6b4bca32de4a538071a652409e93b2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:43.123667 851 leader_election.cc:343] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.124971 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cdb81d024ef8404db7098213f0f7839f (DEFAULT_TABLE table=default.Table_62d807019ce448b9b92b551210892637 [id=a7c2757cd0674517923f5861a904d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.125319 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb81d024ef8404db7098213f0f7839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.128202 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.128713 1013 tablet_service.cc:1468] Processing CreateTablet for tablet ae25051c24024b5897d8115e032ca3b0 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.129050 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae25051c24024b5897d8115e032ca3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.129284 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7a78321046a444bcbe8dd7773b39d6c7 (DEFAULT_TABLE table=default.Table_7744264852c74b1383d4855dcc2f149e [id=1e891425c2c64d44bbefd09310adb93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.129666 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a78321046a444bcbe8dd7773b39d6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.134443 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2cb842878d8c48158a0729047d6b7da2 (DEFAULT_TABLE table=default.Table_d8ffd5ca03a642c699002e8361098a0a [id=b980add8c7b042318afd2417cdc46bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.134876 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cb842878d8c48158a0729047d6b7da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.129290 713 leader_election.cc:343] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.140194 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 749849e1bb19426c97effb4b815e42ad (DEFAULT_TABLE table=default.Table_52c15150adb34323927aadb8acbc21ab [id=079b5c1c6894499eb8278ef1a9883f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.140645 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749849e1bb19426c97effb4b815e42ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.145221 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 595cd3f42a624ed3970d7ba8052a6d60 (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.145671 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595cd3f42a624ed3970d7ba8052a6d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.148051 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d6b4bca32de4a538071a652409e93b2" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.148280 2723 consensus_queue.cc:1035] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:52:43.149291 850 leader_election.cc:343] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.149695 850 leader_election.cc:304] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.150557 2723 raft_consensus.cc:2747] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.152904 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "962633b0d9644c308f0b7fe1db714daf" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.153525 850 leader_election.cc:343] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.153697 850 leader_election.cc:304] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.154032 2861 raft_consensus.cc:2747] T 962633b0d9644c308f0b7fe1db714daf P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.157194 588 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1748e9e9bdbc453da0c3ff120ae70ffd consensus state has no leader
I20250625 07:52:43.157477 2861 raft_consensus.cc:491] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.157586 588 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aa08f96a75714b51bcde3e7b3dcf2c77" } modify_external_catalogs: true
I20250625 07:52:43.157609 2861 raft_consensus.cc:513] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.157738 588 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_aa08f96a75714b51bcde3e7b3dcf2c77" } modify_external_catalogs: true
I20250625 07:52:43.159759 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 1021a582ade04f80ba8e582e5040a217 (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.160176 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021a582ade04f80ba8e582e5040a217. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.160447 584 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:43317, user_credentials={real_user=slave}} blocked reactor thread for 63927.5us
I20250625 07:52:43.160889 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 44bfcc2b20ba4d0b83641988f2c0ee35 (table default.Table_9f2673ddf3e34a969d58c4514580bdcd [id=2aad9533753e4ff9b9c0bab636a8f85b]) successfully deleted
I20250625 07:52:43.161816 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3758b4fc14df40f3bb9a6e77699981ea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.161886 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3758b4fc14df40f3bb9a6e77699981ea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.162353 851 leader_election.cc:343] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.162525 2861 leader_election.cc:290] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.163121 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7a78321046a444bcbe8dd7773b39d6c7 (DEFAULT_TABLE table=default.Table_7744264852c74b1383d4855dcc2f149e [id=1e891425c2c64d44bbefd09310adb93e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:43.163262 850 leader_election.cc:343] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.163370 850 leader_election.cc:304] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.163480 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a78321046a444bcbe8dd7773b39d6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.163734 2861 raft_consensus.cc:2747] T 3758b4fc14df40f3bb9a6e77699981ea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.168216 1161 ts_tablet_manager.cc:1918] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250625 07:52:43.168332 1161 log.cc:1199] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/8cadd41655404316b10939e46d370c76
I20250625 07:52:43.170717 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e0793ddef83409a952052783347dad5" } modify_external_catalogs: true
I20250625 07:52:43.170889 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 07837660a7db47299b8e11f3335cb5d7
I20250625 07:52:43.170892 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_5e0793ddef83409a952052783347dad5" } modify_external_catalogs: true
I20250625 07:52:43.171249 589 catalog_manager.cc:6652] ReplaceTablet: tablet 07837660a7db47299b8e11f3335cb5d7 deleted and replaced by tablet 8a5002ad80bf480299c3e0713f89935c
I20250625 07:52:43.171519 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_feadef2afe4a46b6beb77ab7bba6c789" } modify_external_catalogs: true
I20250625 07:52:43.171653 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_feadef2afe4a46b6beb77ab7bba6c789" } modify_external_catalogs: true
I20250625 07:52:43.172240 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 44906d1dd7a84ec8bcc6701bdd2a1872
I20250625 07:52:43.172602 595 catalog_manager.cc:6652] ReplaceTablet: tablet 44906d1dd7a84ec8bcc6701bdd2a1872 deleted and replaced by tablet c8e9949fdb754d25860c3f1f0fb010a5
I20250625 07:52:43.172926 595 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce15eace6bc5411d808b723c797ed21e" } modify_external_catalogs: true
I20250625 07:52:43.173099 595 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ce15eace6bc5411d808b723c797ed21e" } modify_external_catalogs: true
I20250625 07:52:43.173135 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet eb1718d6cc4b4651b262020d5056b88e
I20250625 07:52:43.173691 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7a175cc57ae741328a3d515859d7a98a" } new_table_name: "default.Table_0c9066d43275486bb4dc05f0cd75636f" modify_external_catalogs: true
W20250625 07:52:43.174940 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 267f9accd9b34154aa0896055a930c73 consensus state has no leader
I20250625 07:52:43.175271 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4706d69e43dd40e7ab4fb5b9fb2725a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.175848 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4706d69e43dd40e7ab4fb5b9fb2725a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.178339 590 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 0cdee1535e5a4a21804e28e78aaea0eb (table default.Table_32aef73480b241f1abf94bdee1fc6fc3 [id=fb8226e9c0a440629bae51d5425f6f31]). Expected version 2 got 0
I20250625 07:52:43.179132 591 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: default.Table_c30b6032c08743888c13c120d548a6a2 [id=29e4ffaed4f148b6b10e38231435c071] alter complete (version 3)
I20250625 07:52:43.187915 740 tablet_service.cc:1468] Processing CreateTablet for tablet 254607ebd73c4f508597b5c6c474f445 (DEFAULT_TABLE table=default.Table_c194642288ef486a9d9aac5a144c385e [id=52729ba430b344c8903802c0a5abc5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.188333 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254607ebd73c4f508597b5c6c474f445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.190354 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 87447ae5f35a44fe9cf5706e306bd5d7 (DEFAULT_TABLE table=default.Table_f4e91d280b644abab2e435312dc3925a [id=4ca429d7af74446a84664e96e2a9b653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.190732 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87447ae5f35a44fe9cf5706e306bd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.192009 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 32a9f4cf6838411fab06fa6d35ae9492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:18 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.192198 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d2bb8b0a8f45079083f2b42bc25011 with delete_type TABLET_DATA_DELETED (replaced by tablet bfb421ea711a461890e267153250acea) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.192413 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa2852e807e461ea1774ab91c170cb8 (DEFAULT_TABLE table=default.Table_9a0629f60a8e4eb999d459ff1d163f72 [id=086d7fc2b013479aac324f8293d4dd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.192482 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 7b112baddf9e4da3902734fc3fe31e9a (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.192802 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa2852e807e461ea1774ab91c170cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.192880 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af43b9ceeb4f2b812ee251ea8545ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.193168 887 tablet_service.cc:1468] Processing CreateTablet for tablet ebd4b665b7b14aef856f53262ce7ae52 (DEFAULT_TABLE table=default.Table_9bda35c7fae042e7af6631324add64d8 [id=b760e637286c4ec39142b21a32d0e6fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.193167 1011 tablet_service.cc:1468] Processing CreateTablet for tablet ff24c012bcdb49d58b031ac77d2fe94d (DEFAULT_TABLE table=default.Table_ab877bb53ad749da82c8b331ed9f4821 [id=ba1d2cd4a37940ce879f9cfbd7975279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.193470 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff24c012bcdb49d58b031ac77d2fe94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.195415 755 tablet_service.cc:1468] Processing CreateTablet for tablet ae25051c24024b5897d8115e032ca3b0 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.197821 590 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_342146f029294bdaa3d7e68bc9e8d6b2" } new_table_name: "default.Table_9497568a43b443579e9f3cffc76f62d7" modify_external_catalogs: true
I20250625 07:52:43.198072 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_548f6eeb0c9046d7a282291743cef43b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.198702 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_548f6eeb0c9046d7a282291743cef43b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.199622 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092" is_pre_election: true
I20250625 07:52:43.198163 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_a237e208ea414db897580bbf7299db3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.199836 914 raft_consensus.cc:2466] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 0.
W20250625 07:52:43.200160 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a237e208ea414db897580bbf7299db3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.200248 588 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 8127db9a35cd46ee99a95fe27ab2454e
I20250625 07:52:43.200232 894 tablet_service.cc:1468] Processing CreateTablet for tablet 254607ebd73c4f508597b5c6c474f445 (DEFAULT_TABLE table=default.Table_c194642288ef486a9d9aac5a144c385e [id=52729ba430b344c8903802c0a5abc5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.200520 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1432) took 2972 ms. Trace:
I20250625 07:52:43.200614 1160 rpcz_store.cc:276] 0625 07:52:40.228382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.228522 (+ 140us) service_pool.cc:225] Handling call
0625 07:52:43.007119 (+2778597us) ts_tablet_manager.cc:1145] Deleting tablet cd226579c77b4938bb031fac0412cbb1
0625 07:52:43.200315 (+193196us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.200502 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2778366}
I20250625 07:52:43.201047 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1433) took 2958 ms. Trace:
I20250625 07:52:43.201119 1160 rpcz_store.cc:276] 0625 07:52:40.242191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.243337 (+ 1146us) service_pool.cc:225] Handling call
0625 07:52:43.200923 (+2957586us) ts_tablet_manager.cc:1145] Deleting tablet f64c42861e60451e92131a435f95b012
0625 07:52:43.200933 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.201032 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2957385}
I20250625 07:52:43.201442 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1434) took 2813 ms. Trace:
I20250625 07:52:43.201514 1160 rpcz_store.cc:276] 0625 07:52:40.387600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.387657 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:43.201315 (+2813658us) ts_tablet_manager.cc:1145] Deleting tablet d320734990404d1e9222bea1d88a18fe
0625 07:52:43.201325 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.201425 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2813446}
I20250625 07:52:43.201745 1160 tablet_replica.cc:331] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:43.201979 1160 raft_consensus.cc:2241] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.202136 1160 raft_consensus.cc:2270] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.202659 1160 ts_tablet_manager.cc:1905] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.199846 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 896b9a3fe00a4dc491cb4de0d5201c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.207629 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039992783bb147178880d0b77aeced0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.207793 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 254607ebd73c4f508597b5c6c474f445. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.207974 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e66aff696c084e379578b9350e4ac23d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.208791 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1da59ad9c6f14064b71581e08cacf7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.213263 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0f34ff108d480d87b95ffa135df86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.214341 894 tablet_service.cc:1468] Processing CreateTablet for tablet cdb81d024ef8404db7098213f0f7839f (DEFAULT_TABLE table=default.Table_62d807019ce448b9b92b551210892637 [id=a7c2757cd0674517923f5861a904d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.214468 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 3823279d7e06435484f1b82836ebcf39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:41 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.214712 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb81d024ef8404db7098213f0f7839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.214704 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 07837660a7db47299b8e11f3335cb5d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a5002ad80bf480299c3e0713f89935c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.215584 744 tablet_service.cc:1468] Processing CreateTablet for tablet 50fadc50d91649aeaff0036ce4ffdeeb (DEFAULT_TABLE table=default.Table_cbe74ec8a2f443a4afe33052ace1ce64 [id=69688e8d49e04e4f97a2c9a8279c65bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.216796 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c175adca5b37414f9da37047d232e973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.216825 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b112baddf9e4da3902734fc3fe31e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.217056 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.217643 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e9949fdb754d25860c3f1f0fb010a5) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.218624 742 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa2852e807e461ea1774ab91c170cb8 (DEFAULT_TABLE table=default.Table_9a0629f60a8e4eb999d459ff1d163f72 [id=086d7fc2b013479aac324f8293d4dd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.219180 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a98db126a0084fe7bd96fbd87557a7a8 (DEFAULT_TABLE table=default.Table_8ba515bb0f4d404da06248724a562452 [id=5985ca5f897b49618b75b51c34c19813]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:43.219260 713 consensus_peers.cc:487] T 8cadd41655404316b10939e46d370c76 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:43.219619 713 consensus_peers.cc:487] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:43.219820 747 tablet_service.cc:1515] Processing DeleteTablet for tablet eb1718d6cc4b4651b262020d5056b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220006 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 51c584e6bffb4f0a8020dadf78d718a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220135 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b04a575e0984f879f0179c8baa3f3fa with delete_type TABLET_DATA_DELETED (replaced by tablet fed43adfae594e3b91ed1012838c3317) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220242 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 9578d77db00242d48f4e8f6f8263ef09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220243 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98db126a0084fe7bd96fbd87557a7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.220352 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f7240577a874a59b3268ff7f0c1beec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220451 747 tablet_service.cc:1515] Processing DeleteTablet for tablet d0a75906385744e5ba6938f451a2d7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:30 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220536 747 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bc347c278c44c69444caf9ded72e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 9b66fad15471497390fc753b4015bb8f) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220643 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e06ecad6c7464ffe8ec6f632634307c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 328b21f669c442fb86259500e794d222) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220811 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 506794253c84486faa926de885c48c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:22 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.220947 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 484a49fab6134d059c64f5a3f2b29e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.221202 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f8ae5b56ef49e594befd1caec5b049. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.224828 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a78321046a444bcbe8dd7773b39d6c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.225015 1163 tablet_bootstrap.cc:492] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:43.225419 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12db6e59e76f429f9be71ccc77c65d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.226614 1163 tablet_bootstrap.cc:654] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.227818 758 tablet_service.cc:1468] Processing CreateTablet for tablet 595cd3f42a624ed3970d7ba8052a6d60 (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.228550 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2db5d25775ec48da8511146ac7df4068. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.228982 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa8b405985e4331bbdfa5c6d5f6518c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.229574 1017 tablet_service.cc:1468] Processing CreateTablet for tablet fe98fa4e2b794851878d1af9a714e20b (DEFAULT_TABLE table=default.Table_e84a089995ef4ad78eb9025893a5683d [id=007b44f673e646f3bc9a88e30d982a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.229965 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9 (DEFAULT_TABLE table=default.Table_bb21510ec96d482f82b38d009b3d0713 [id=58a0cb0e4c504970aa8db9d3d4ad7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.230283 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe98fa4e2b794851878d1af9a714e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.231166 753 tablet_service.cc:1468] Processing CreateTablet for tablet cdb81d024ef8404db7098213f0f7839f (DEFAULT_TABLE table=default.Table_62d807019ce448b9b92b551210892637 [id=a7c2757cd0674517923f5861a904d8d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.232957 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2ae888c90cc643808b4fdfb3c1da2b75 (DEFAULT_TABLE table=default.Table_f2c190af167a46ca8ba93fe63cc77209 [id=607bfb8bcb0c436da7f02a61739b41c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.233232 756 tablet_service.cc:1468] Processing CreateTablet for tablet 749849e1bb19426c97effb4b815e42ad (DEFAULT_TABLE table=default.Table_52c15150adb34323927aadb8acbc21ab [id=079b5c1c6894499eb8278ef1a9883f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.233634 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1021a582ade04f80ba8e582e5040a217 (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.234162 745 tablet_service.cc:1468] Processing CreateTablet for tablet d8f87840eaad4cf7a6f9c5101b114fbe (DEFAULT_TABLE table=default.Table_44ede952c58346e98591c6e14c6ff095 [id=ee7d0ff485334c3481eb6cf96f2d0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.234537 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet eb1718d6cc4b4651b262020d5056b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.235267 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 50fadc50d91649aeaff0036ce4ffdeeb (DEFAULT_TABLE table=default.Table_cbe74ec8a2f443a4afe33052ace1ce64 [id=69688e8d49e04e4f97a2c9a8279c65bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.234162 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7b112baddf9e4da3902734fc3fe31e9a (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.237237 750 tablet_service.cc:1468] Processing CreateTablet for tablet a98db126a0084fe7bd96fbd87557a7a8 (DEFAULT_TABLE table=default.Table_8ba515bb0f4d404da06248724a562452 [id=5985ca5f897b49618b75b51c34c19813]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.237531 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e9949fdb754d25860c3f1f0fb010a5) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.240276 895 tablet_service.cc:1468] Processing CreateTablet for tablet ae25051c24024b5897d8115e032ca3b0 (DEFAULT_TABLE table=default.Table_e6935ea883d54dd8bc633cdee1530352 [id=94becb97af574a44bfcb6a2b128b1d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.240666 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae25051c24024b5897d8115e032ca3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.241303 713 leader_election.cc:304] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.241253 2718 consensus_queue.cc:1035] T bee1c7f7746a4d6fb763aedadc861421 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:43.241497 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 07837660a7db47299b8e11f3335cb5d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a5002ad80bf480299c3e0713f89935c) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.241725 2718 raft_consensus.cc:2802] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:52:43.241816 2718 raft_consensus.cc:491] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:52:43.241869 2718 raft_consensus.cc:3058] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:43.244449 2718 raft_consensus.cc:513] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.245028 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9759b06b076d4d3aba740791256e8991. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.245754 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f64c42861e60451e92131a435f95b012 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f64c42861e60451e92131a435f95b012 already in progress: creating tablet
I20250625 07:52:43.246490 2718 leader_election.cc:290] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
W20250625 07:52:43.246490 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d320734990404d1e9222bea1d88a18fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d320734990404d1e9222bea1d88a18fe already in progress: creating tablet
I20250625 07:52:43.246529 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1021a582ade04f80ba8e582e5040a217 (DEFAULT_TABLE table=default.Table_48759d4fd35e40ef80e2f1970c37738a [id=b10388c720ac4ea0916d40a49bb5a0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.247253 2811 raft_consensus.cc:491] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.247326 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2851494cfc44b09f0f248b51a11858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.247408 2811 raft_consensus.cc:513] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.247814 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a9c299a4bf4567a6dfaaaf346556c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.247853 892 tablet_service.cc:1468] Processing CreateTablet for tablet 595cd3f42a624ed3970d7ba8052a6d60 (DEFAULT_TABLE table=default.Table_e163a274311f4f969ff05cc5a7685b00 [id=e781cd12ae6e4d0e8efe61c41e60da5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.248203 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2fba569e445d498780d862197d3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.248237 2811 leader_election.cc:290] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.248575 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da1f526d6e3c48a58b34cb1ef0753876. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.248976 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cb842878d8c48158a0729047d6b7da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.249115 986 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 9bccda893fd5466886a4d253172309e1
I20250625 07:52:43.249339 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad9ed79cca34fe8b3b7c15c213e5dbb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.250837 985 leader_election.cc:343] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9bccda893fd5466886a4d253172309e1
I20250625 07:52:43.250942 985 leader_election.cc:304] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:43.251144 2811 raft_consensus.cc:2747] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.251767 878 tablet_service.cc:1468] Processing CreateTablet for tablet 749849e1bb19426c97effb4b815e42ad (DEFAULT_TABLE table=default.Table_52c15150adb34323927aadb8acbc21ab [id=079b5c1c6894499eb8278ef1a9883f19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.252080 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749849e1bb19426c97effb4b815e42ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.257176 2662 webserver.cc:469] Webserver started at http://127.0.104.61:34053/ using document root <none> and password file <none>
I20250625 07:52:43.258361 2662 fs_manager.cc:362] Metadata directory not provided
I20250625 07:52:43.258500 2662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:52:43.261041 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021a582ade04f80ba8e582e5040a217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.261511 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595cd3f42a624ed3970d7ba8052a6d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.267347 1024 tablet_service.cc:1468] Processing CreateTablet for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc (DEFAULT_TABLE table=default.Table_969c5b5b88be48d9bf2fbde2ba3a016e [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.276780 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429026e8d98d46339db143f84897a092"
I20250625 07:52:43.276988 910 raft_consensus.cc:3058] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:43.279024 888 tablet_service.cc:1468] Processing CreateTablet for tablet d8f87840eaad4cf7a6f9c5101b114fbe (DEFAULT_TABLE table=default.Table_44ede952c58346e98591c6e14c6ff095 [id=ee7d0ff485334c3481eb6cf96f2d0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.279403 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f87840eaad4cf7a6f9c5101b114fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.279780 910 raft_consensus.cc:2466] T e35d77471f7d4866a8c30324b5da0170 P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191b611b772a4fda9d1cba98776138a7 in term 1.
I20250625 07:52:43.284561 876 tablet_service.cc:1468] Processing CreateTablet for tablet c04a02358a514349b3923b9603424b2c (DEFAULT_TABLE table=default.Table_84f144f01ac148138bdb55f29f46a4fc [id=c605bf122c454abe9dc08c221f2125f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.284966 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04a02358a514349b3923b9603424b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.287794 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e35d77471f7d4866a8c30324b5da0170" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513"
I20250625 07:52:43.288810 1011 tablet_service.cc:1468] Processing CreateTablet for tablet a43f7951598342d8a1b08f03a20504b5 (DEFAULT_TABLE table=default.Table_7918799ea32a4b6bbd7c95a79962e5f9 [id=d30b1bcdb620435e909fc12f1b9a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:43.289661 713 leader_election.cc:343] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.290539 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fadc50d91649aeaff0036ce4ffdeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.291416 1158 tablet_bootstrap.cc:654] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.297226 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa2852e807e461ea1774ab91c170cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.297889 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b112baddf9e4da3902734fc3fe31e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.298298 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98db126a0084fe7bd96fbd87557a7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.298698 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749849e1bb19426c97effb4b815e42ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.299132 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021a582ade04f80ba8e582e5040a217. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.299506 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595cd3f42a624ed3970d7ba8052a6d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.299876 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb81d024ef8404db7098213f0f7839f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.300262 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f87840eaad4cf7a6f9c5101b114fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.300925 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fadc50d91649aeaff0036ce4ffdeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.301959 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e87498dd77dd4aa9be72930fe7358c42 (DEFAULT_TABLE table=default.Table_b495ebc15ab14b7490f58c02290de736 [id=3f75adb098004d35b0020afe0686a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.302387 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87498dd77dd4aa9be72930fe7358c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.304073 2811 raft_consensus.cc:491] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.304188 2811 raft_consensus.cc:513] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.304787 2811 leader_election.cc:290] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.305305 1160 ts_tablet_manager.cc:1918] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.305398 1160 log.cc:1199] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/6d6b4bca32de4a538071a652409e93b2
I20250625 07:52:43.305712 1160 ts_tablet_manager.cc:1939] T 6d6b4bca32de4a538071a652409e93b2 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:43.305711 986 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:43.306408 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1440) took 2758 ms. Trace:
I20250625 07:52:43.306484 1160 rpcz_store.cc:276] 0625 07:52:40.547719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.547776 (+ 57us) service_pool.cc:225] Handling call
0625 07:52:43.201706 (+2653930us) ts_tablet_manager.cc:1145] Deleting tablet 6d6b4bca32de4a538071a652409e93b2
0625 07:52:43.201714 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.275096 (+ 73382us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.305202 (+ 30106us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.306388 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653683}
I20250625 07:52:43.307942 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 97457191e3f04941b13e0a815dc22588 (DEFAULT_TABLE table=default.Table_a648db116d7d43969d0cf3d5c2501450 [id=a0460709e23d4e89b13be8e12c759e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.308266 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97457191e3f04941b13e0a815dc22588. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.308840 1012 tablet_service.cc:1468] Processing CreateTablet for tablet c04a02358a514349b3923b9603424b2c (DEFAULT_TABLE table=default.Table_84f144f01ac148138bdb55f29f46a4fc [id=c605bf122c454abe9dc08c221f2125f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.309150 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04a02358a514349b3923b9603424b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.319643 739 tablet_service.cc:1468] Processing CreateTablet for tablet 87447ae5f35a44fe9cf5706e306bd5d7 (DEFAULT_TABLE table=default.Table_f4e91d280b644abab2e435312dc3925a [id=4ca429d7af74446a84664e96e2a9b653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.320266 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebd4b665b7b14aef856f53262ce7ae52. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.323392 985 leader_election.cc:343] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 2bcfe7c3c54741fbb0aca69650fc8eee
I20250625 07:52:43.323630 985 leader_election.cc:304] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:43.323916 2811 raft_consensus.cc:2747] T 2bcfe7c3c54741fbb0aca69650fc8eee P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.326866 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.327286 748 tablet_service.cc:1468] Processing CreateTablet for tablet 2ae888c90cc643808b4fdfb3c1da2b75 (DEFAULT_TABLE table=default.Table_f2c190af167a46ca8ba93fe63cc77209 [id=607bfb8bcb0c436da7f02a61739b41c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.327452 746 tablet_service.cc:1468] Processing CreateTablet for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc (DEFAULT_TABLE table=default.Table_969c5b5b88be48d9bf2fbde2ba3a016e [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.327286 751 tablet_service.cc:1468] Processing CreateTablet for tablet fe98fa4e2b794851878d1af9a714e20b (DEFAULT_TABLE table=default.Table_e84a089995ef4ad78eb9025893a5683d [id=007b44f673e646f3bc9a88e30d982a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.327772 741 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9 (DEFAULT_TABLE table=default.Table_bb21510ec96d482f82b38d009b3d0713 [id=58a0cb0e4c504970aa8db9d3d4ad7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.327565 752 tablet_service.cc:1468] Processing CreateTablet for tablet ff24c012bcdb49d58b031ac77d2fe94d (DEFAULT_TABLE table=default.Table_ab877bb53ad749da82c8b331ed9f4821 [id=ba1d2cd4a37940ce879f9cfbd7975279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.328868 1162 tablet_bootstrap.cc:492] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:43.329012 1162 ts_tablet_manager.cc:1397] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.341s user 0.008s sys 0.003s
I20250625 07:52:43.329819 1162 raft_consensus.cc:357] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.330009 1162 raft_consensus.cc:383] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.330065 1162 raft_consensus.cc:738] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.330268 1162 consensus_queue.cc:260] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.330742 1162 ts_tablet_manager.cc:1428] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250625 07:52:43.331569 1163 tablet_bootstrap.cc:492] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:43.335393 754 tablet_service.cc:1468] Processing CreateTablet for tablet c04a02358a514349b3923b9603424b2c (DEFAULT_TABLE table=default.Table_84f144f01ac148138bdb55f29f46a4fc [id=c605bf122c454abe9dc08c221f2125f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.339329 747 tablet_service.cc:1468] Processing CreateTablet for tablet a43f7951598342d8a1b08f03a20504b5 (DEFAULT_TABLE table=default.Table_7918799ea32a4b6bbd7c95a79962e5f9 [id=d30b1bcdb620435e909fc12f1b9a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.340247 1158 tablet_bootstrap.cc:492] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.340375 1158 ts_tablet_manager.cc:1397] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.277s user 0.003s sys 0.000s
I20250625 07:52:43.341090 1158 raft_consensus.cc:357] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.341255 1158 raft_consensus.cc:383] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.341300 1158 raft_consensus.cc:738] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.341487 1158 consensus_queue.cc:260] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.342283 2869 raft_consensus.cc:491] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.342396 2869 raft_consensus.cc:513] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.342942 2869 leader_election.cc:290] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.346412 1161 ts_tablet_manager.cc:1939] T 8cadd41655404316b10939e46d370c76 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:43.349205 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1499) took 1092 ms. Trace:
I20250625 07:52:43.349303 1161 rpcz_store.cc:276] 0625 07:52:42.257256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:42.257317 (+ 61us) service_pool.cc:225] Handling call
0625 07:52:42.953741 (+696424us) ts_tablet_manager.cc:1145] Deleting tablet 8cadd41655404316b10939e46d370c76
0625 07:52:42.953752 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.164816 (+211064us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.168116 (+ 3300us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.349183 (+181067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":696234}
I20250625 07:52:43.349749 1161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47274 (request call id 1500) took 1091 ms. Trace:
I20250625 07:52:43.349833 1161 rpcz_store.cc:276] 0625 07:52:42.257910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:42.257966 (+ 56us) service_pool.cc:225] Handling call
0625 07:52:43.349619 (+1091653us) ts_tablet_manager.cc:1145] Deleting tablet 0009ea8aaf6e4fdd8ff602bd59968de3
0625 07:52:43.349630 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.349732 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1091477}
I20250625 07:52:43.349936 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 8cadd41655404316b10939e46d370c76 (table default.Table_f6ad13e139c64d20abeb28e8584f0781 [id=03e6478224ea4f819a9533dd713b3bc5]) successfully deleted
I20250625 07:52:43.350616 1161 tablet_replica.cc:331] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
W20250625 07:52:43.350745 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 0009ea8aaf6e4fdd8ff602bd59968de3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0009ea8aaf6e4fdd8ff602bd59968de3 already in progress: creating tablet
I20250625 07:52:43.350793 1161 raft_consensus.cc:2241] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.350943 1161 raft_consensus.cc:2270] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:43.351117 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d6a57b2d0d74ad98fb776f7a8e67ef1 already in progress: creating tablet
W20250625 07:52:43.351423 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4b2e211834bf4a1591759658e0f8e2be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b2e211834bf4a1591759658e0f8e2be already in progress: creating tablet
I20250625 07:52:43.351457 1161 ts_tablet_manager.cc:1905] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:43.351704 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 4718c51c79ac41cebfca7dcc567ddecd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4718c51c79ac41cebfca7dcc567ddecd already in progress: creating tablet
W20250625 07:52:43.351999 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 535259cb581843568b1561c607a7d542 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 535259cb581843568b1561c607a7d542 already in progress: creating tablet
W20250625 07:52:43.352283 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 3f52c6475a644519828e7947e3dd46b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f52c6475a644519828e7947e3dd46b1 already in progress: creating tablet
I20250625 07:52:43.353140 1158 ts_tablet_manager.cc:1428] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250625 07:52:43.360407 1158 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.362644 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 6d6b4bca32de4a538071a652409e93b2 (table default.Table_7fb5f2a41fbf4e11add0bf364fc3628e [id=cd272991373c4fee95c14b59535eaa84]) successfully deleted
I20250625 07:52:43.363433 1161 ts_tablet_manager.cc:1918] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.363564 1161 log.cc:1199] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/9bccda893fd5466886a4d253172309e1
I20250625 07:52:43.363857 1161 ts_tablet_manager.cc:1939] T 9bccda893fd5466886a4d253172309e1 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:43.363901 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f7240577a874a59b3268ff7f0c1beec" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.362393 713 leader_election.cc:304] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092; no voters: 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.364535 2869 raft_consensus.cc:2802] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:52:43.365303 1158 tablet_bootstrap.cc:654] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.365365 2869 raft_consensus.cc:695] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [term 1 LEADER]: Becoming Leader. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Running, Role: LEADER
I20250625 07:52:43.365907 2869 consensus_queue.cc:237] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.368552 1161 tablet_replica.cc:331] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:43.368772 1161 raft_consensus.cc:2241] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.368943 1161 raft_consensus.cc:2270] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.369380 1158 tablet_bootstrap.cc:492] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.369711 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 9bccda893fd5466886a4d253172309e1 (table default.Table_3910e10f6f4b41eeabfd11ac6c4343c7 [id=c90351757d464813b5d0d67936dd69c5]) successfully deleted
I20250625 07:52:43.370357 1158 ts_tablet_manager.cc:1397] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250625 07:52:43.369467 1161 ts_tablet_manager.cc:1905] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:43.370664 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet fe3ccad4a5a34bfba36c5c18ec5038c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe3ccad4a5a34bfba36c5c18ec5038c0 already in progress: creating tablet
I20250625 07:52:43.371944 2662 fs_manager.cc:714] Time spent opening directory manager: real 0.035s user 0.004s sys 0.005s
I20250625 07:52:43.379186 1158 raft_consensus.cc:357] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.379406 1158 raft_consensus.cc:383] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.379477 1158 raft_consensus.cc:738] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.379690 1158 consensus_queue.cc:260] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:43.380157 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet 6c8893ee33bd4f88ac5c913e99fb4262 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c8893ee33bd4f88ac5c913e99fb4262 already in progress: creating tablet
W20250625 07:52:43.381117 583 catalog_manager.cc:4908] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): delete failed for tablet a8f25febb41c403ca8be268206157756 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8f25febb41c403ca8be268206157756 already in progress: creating tablet
W20250625 07:52:43.382686 713 consensus_peers.cc:487] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:43.382858 713 leader_election.cc:343] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.384176 2875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:52:43.387112 2662 fs_manager.cc:730] Time spent opening block manager: real 0.013s user 0.007s sys 0.003s
I20250625 07:52:43.388186 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae25051c24024b5897d8115e032ca3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.388538 1158 ts_tablet_manager.cc:1428] T 61e076870efc4744b1224ddcb0eef3fa P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250625 07:52:43.387358 2662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
uuid: "789c06e7de2842a1b40e97e0778ab6c9"
format_stamp: "Formatted at 2025-06-25 07:52:08 on dist-test-slave-ggw5"
I20250625 07:52:43.389360 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87447ae5f35a44fe9cf5706e306bd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.331516 890 tablet_service.cc:1468] Processing CreateTablet for tablet 87447ae5f35a44fe9cf5706e306bd5d7 (DEFAULT_TABLE table=default.Table_f4e91d280b644abab2e435312dc3925a [id=4ca429d7af74446a84664e96e2a9b653]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.389798 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87447ae5f35a44fe9cf5706e306bd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.389817 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ae888c90cc643808b4fdfb3c1da2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.390225 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.390486 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff24c012bcdb49d58b031ac77d2fe94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.332240 886 tablet_service.cc:1468] Processing CreateTablet for tablet a98db126a0084fe7bd96fbd87557a7a8 (DEFAULT_TABLE table=default.Table_8ba515bb0f4d404da06248724a562452 [id=5985ca5f897b49618b75b51c34c19813]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.332432 884 tablet_service.cc:1515] Processing DeleteTablet for tablet eb1718d6cc4b4651b262020d5056b88e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.391788 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98db126a0084fe7bd96fbd87557a7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.392796 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe98fa4e2b794851878d1af9a714e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.392979 884 tablet_service.cc:1468] Processing CreateTablet for tablet fe98fa4e2b794851878d1af9a714e20b (DEFAULT_TABLE table=default.Table_e84a089995ef4ad78eb9025893a5683d [id=007b44f673e646f3bc9a88e30d982a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.332455 891 tablet_service.cc:1515] Processing DeleteTablet for tablet daf115dd3aa548dc802ed0362c274d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.394356 891 tablet_service.cc:1468] Processing CreateTablet for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc (DEFAULT_TABLE table=default.Table_969c5b5b88be48d9bf2fbde2ba3a016e [id=9f2c3c35f2ed4bbf91bd5d4cd3a6b5cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.332674 883 tablet_service.cc:1468] Processing CreateTablet for tablet 0fa2852e807e461ea1774ab91c170cb8 (DEFAULT_TABLE table=default.Table_9a0629f60a8e4eb999d459ff1d163f72 [id=086d7fc2b013479aac324f8293d4dd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.396811 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.332890 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 44906d1dd7a84ec8bcc6701bdd2a1872 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e9949fdb754d25860c3f1f0fb010a5) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.332948 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b19f6c181ee4e24883b0d7df822ab3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:20 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.397181 880 tablet_service.cc:1468] Processing CreateTablet for tablet a43f7951598342d8a1b08f03a20504b5 (DEFAULT_TABLE table=default.Table_7918799ea32a4b6bbd7c95a79962e5f9 [id=d30b1bcdb620435e909fc12f1b9a9ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.397315 877 tablet_service.cc:1468] Processing CreateTablet for tablet 508d04fb4c0b4f2b91531b197660c072 (DEFAULT_TABLE table=default.Table_bdce2b3069f1428487769ca6ac55ff24 [id=0aa1bf3dd031411b9b8c771a058b7c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.332954 885 tablet_service.cc:1468] Processing CreateTablet for tablet 7b112baddf9e4da3902734fc3fe31e9a (DEFAULT_TABLE table=default.Table_83a512e0c6d648cc857a484855ea1264 [id=7a5b6b2d13bb4a188e5827e60c90a604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.333312 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a80d81cc2d524f318b858a9bd97a89b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.398262 893 tablet_service.cc:1468] Processing CreateTablet for tablet e87498dd77dd4aa9be72930fe7358c42 (DEFAULT_TABLE table=default.Table_b495ebc15ab14b7490f58c02290de736 [id=3f75adb098004d35b0020afe0686a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.333411 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5a40faa418420ea31bc1d46cab1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:21 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.398711 882 tablet_service.cc:1468] Processing CreateTablet for tablet 97457191e3f04941b13e0a815dc22588 (DEFAULT_TABLE table=default.Table_a648db116d7d43969d0cf3d5c2501450 [id=a0460709e23d4e89b13be8e12c759e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.333650 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8127db9a35cd46ee99a95fe27ab2454e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.332736 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b8b5bb3fe3ef4bfdba93b862784a9f61 with delete_type TABLET_DATA_DELETED (replaced by tablet bc387934e5204a3e9ce6f13d1dc3f777) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.334959 878 tablet_service.cc:1468] Processing CreateTablet for tablet ff24c012bcdb49d58b031ac77d2fe94d (DEFAULT_TABLE table=default.Table_ab877bb53ad749da82c8b331ed9f4821 [id=ba1d2cd4a37940ce879f9cfbd7975279]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.399256 1158 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.335615 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 07837660a7db47299b8e11f3335cb5d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a5002ad80bf480299c3e0713f89935c) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.340996 892 tablet_service.cc:1468] Processing CreateTablet for tablet 50fadc50d91649aeaff0036ce4ffdeeb (DEFAULT_TABLE table=default.Table_cbe74ec8a2f443a4afe33052ace1ce64 [id=69688e8d49e04e4f97a2c9a8279c65bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.341312 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2ae888c90cc643808b4fdfb3c1da2b75 (DEFAULT_TABLE table=default.Table_f2c190af167a46ca8ba93fe63cc77209 [id=607bfb8bcb0c436da7f02a61739b41c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250625 07:52:43.363893 850 consensus_peers.cc:487] T 34544157495f4e14a7978e0eec70b522 P 429026e8d98d46339db143f84897a092 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:43.400942 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fa2852e807e461ea1774ab91c170cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.401373 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ae888c90cc643808b4fdfb3c1da2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.401710 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fadc50d91649aeaff0036ce4ffdeeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.402062 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff24c012bcdb49d58b031ac77d2fe94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.402427 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87498dd77dd4aa9be72930fe7358c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.402807 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f7951598342d8a1b08f03a20504b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.403793 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b112baddf9e4da3902734fc3fe31e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.404176 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508d04fb4c0b4f2b91531b197660c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.404508 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.409682 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1437) took 2867 ms. Trace:
I20250625 07:52:43.409778 1160 rpcz_store.cc:276] 0625 07:52:40.541904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.541976 (+ 72us) service_pool.cc:225] Handling call
0625 07:52:43.306760 (+2764784us) ts_tablet_manager.cc:1145] Deleting tablet 1bbf1cfac4e14a20b3fb53607cdf100f
0625 07:52:43.306769 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.409650 (+102881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758226}
I20250625 07:52:43.410125 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1471) took 2444 ms. Trace:
I20250625 07:52:43.410192 1160 rpcz_store.cc:276] 0625 07:52:40.965936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.986824 (+ 20888us) service_pool.cc:225] Handling call
0625 07:52:43.410011 (+2423187us) ts_tablet_manager.cc:1145] Deleting tablet a642a958550a40a48453f47d01b062a2
0625 07:52:43.410021 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.410109 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423002}
I20250625 07:52:43.410461 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1472) took 2444 ms. Trace:
I20250625 07:52:43.410764 1163 ts_tablet_manager.cc:1397] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.186s user 0.006s sys 0.001s
W20250625 07:52:43.412228 713 leader_election.cc:343] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 8f7240577a874a59b3268ff7f0c1beec
I20250625 07:52:43.412350 713 leader_election.cc:304] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.412590 2876 raft_consensus.cc:2747] T 8f7240577a874a59b3268ff7f0c1beec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.413060 2876 consensus_queue.cc:1035] T 7d930a09f5e542aab943a51072f0c626 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:43.413406 2723 raft_consensus.cc:491] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.413525 2723 raft_consensus.cc:513] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.414220 2723 leader_election.cc:290] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
W20250625 07:52:43.414974 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 1bbf1cfac4e14a20b3fb53607cdf100f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bbf1cfac4e14a20b3fb53607cdf100f already in progress: creating tablet
W20250625 07:52:43.415236 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a642a958550a40a48453f47d01b062a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a642a958550a40a48453f47d01b062a2 already in progress: creating tablet
I20250625 07:52:43.415729 2612 raft_consensus.cc:491] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.415880 2612 raft_consensus.cc:513] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.416440 2612 leader_election.cc:290] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429026e8d98d46339db143f84897a092 (127.0.104.2:45317), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:43.417200 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0a75906385744e5ba6938f451a2d7ec" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.417538 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b19f6c181ee4e24883b0d7df822ab3a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.418852 713 leader_election.cc:343] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.418953 1158 tablet_bootstrap.cc:654] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.419250 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9e58c25a684cc4bd1bf4b3fe2fa9dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.420804 1161 ts_tablet_manager.cc:1918] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.420912 1161 log.cc:1199] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/32a9f4cf6838411fab06fa6d35ae9492
I20250625 07:52:43.421044 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97457191e3f04941b13e0a815dc22588. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.421224 1161 ts_tablet_manager.cc:1939] T 32a9f4cf6838411fab06fa6d35ae9492 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
W20250625 07:52:43.421783 713 consensus_peers.cc:487] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:52:43.422009 713 consensus_peers.cc:487] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 -> Peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Couldn't send request to peer 8f2cdaf87b9d4b2a919098bf65efd513. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:52:43.423276 1158 tablet_bootstrap.cc:492] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.423409 1158 ts_tablet_manager.cc:1397] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20250625 07:52:43.424079 1158 raft_consensus.cc:357] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.424234 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d8f87840eaad4cf7a6f9c5101b114fbe (DEFAULT_TABLE table=default.Table_44ede952c58346e98591c6e14c6ff095 [id=ee7d0ff485334c3481eb6cf96f2d0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.424603 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.424661 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 508d04fb4c0b4f2b91531b197660c072 (DEFAULT_TABLE table=default.Table_bdce2b3069f1428487769ca6ac55ff24 [id=0aa1bf3dd031411b9b8c771a058b7c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.425215 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 32a9f4cf6838411fab06fa6d35ae9492 (table default.Table_2560f175e7af433db12e4a7c5b4c3c69 [id=930d611810af48fdb6295ea294faa440]) successfully deleted
I20250625 07:52:43.425401 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe98fa4e2b794851878d1af9a714e20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.424245 1158 raft_consensus.cc:383] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.426836 1158 raft_consensus.cc:738] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.427273 1158 consensus_queue.cc:260] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.428817 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f7951598342d8a1b08f03a20504b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.432950 1158 ts_tablet_manager.cc:1428] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250625 07:52:43.444619 1161 tablet_replica.cc:331] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:43.410514 1160 rpcz_store.cc:276] 0625 07:52:40.966012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.987018 (+ 21006us) service_pool.cc:225] Handling call
0625 07:52:43.410376 (+2423358us) ts_tablet_manager.cc:1145] Deleting tablet a571983779854eb398b3b0b42c34f83a
0625 07:52:43.410384 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.410450 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423258}
I20250625 07:52:43.444864 1161 raft_consensus.cc:2241] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.444940 1158 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.444974 1160 tablet_replica.cc:331] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:43.445025 1161 raft_consensus.cc:2270] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.445155 1160 raft_consensus.cc:2241] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.445288 1160 raft_consensus.cc:2270] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.445538 1161 ts_tablet_manager.cc:1905] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.445757 1160 ts_tablet_manager.cc:1905] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.446240 1158 tablet_bootstrap.cc:654] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.385274 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9 (DEFAULT_TABLE table=default.Table_bb21510ec96d482f82b38d009b3d0713 [id=58a0cb0e4c504970aa8db9d3d4ad7830]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.450096 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bcef19b1ecc4c4db9d9070e6a6ae4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.452090 743 tablet_service.cc:1468] Processing CreateTablet for tablet 508d04fb4c0b4f2b91531b197660c072 (DEFAULT_TABLE table=default.Table_bdce2b3069f1428487769ca6ac55ff24 [id=0aa1bf3dd031411b9b8c771a058b7c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.452925 592 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet c4d3bff930cf4ce7941d2a9608f889b6
I20250625 07:52:43.453220 778 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b19f6c181ee4e24883b0d7df822ab3a" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.454521 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c04a02358a514349b3923b9603424b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.454547 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.454738 749 tablet_service.cc:1468] Processing CreateTablet for tablet e87498dd77dd4aa9be72930fe7358c42 (DEFAULT_TABLE table=default.Table_b495ebc15ab14b7490f58c02290de736 [id=3f75adb098004d35b0020afe0686a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.455049 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87498dd77dd4aa9be72930fe7358c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.455721 589 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet e7e139c6da5a41c6a4a754f5511899d7
I20250625 07:52:43.457190 742 tablet_service.cc:1468] Processing CreateTablet for tablet 97457191e3f04941b13e0a815dc22588 (DEFAULT_TABLE table=default.Table_a648db116d7d43969d0cf3d5c2501450 [id=a0460709e23d4e89b13be8e12c759e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:52:43.457612 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97457191e3f04941b13e0a815dc22588. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.459834 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508d04fb4c0b4f2b91531b197660c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.462273 1158 tablet_bootstrap.cc:492] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.462397 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ae888c90cc643808b4fdfb3c1da2b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.462574 1158 ts_tablet_manager.cc:1397] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20250625 07:52:43.463317 1158 raft_consensus.cc:357] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.463559 1158 raft_consensus.cc:383] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.463652 1158 raft_consensus.cc:738] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.463930 1158 consensus_queue.cc:260] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.465649 1158 ts_tablet_manager.cc:1428] T fe630469fe7c4f4bac28277896eddc9c P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250625 07:52:43.389006 2662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:52:43.467375 1162 tablet_bootstrap.cc:492] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:43.468384 1158 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
W20250625 07:52:43.455112 851 leader_election.cc:343] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.469738 1158 tablet_bootstrap.cc:654] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.474608 1162 tablet_bootstrap.cc:654] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.475808 2407 ts_tablet_manager.cc:1918] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.475924 2407 log.cc:1199] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/065822c50586488785f7d9d53841a4dd
I20250625 07:52:43.475808 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508d04fb4c0b4f2b91531b197660c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.476227 2407 ts_tablet_manager.cc:1939] T 065822c50586488785f7d9d53841a4dd P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:43.478682 1161 ts_tablet_manager.cc:1918] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.478811 1161 log.cc:1199] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/26d2bb8b0a8f45079083f2b42bc25011
I20250625 07:52:43.479187 1161 ts_tablet_manager.cc:1939] T 26d2bb8b0a8f45079083f2b42bc25011 P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:43.464612 742 tablet_service.cc:1515] Processing DeleteTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.480799 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f7951598342d8a1b08f03a20504b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:52:43.485051 2811 raft_consensus.cc:491] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.485177 2811 raft_consensus.cc:513] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.485705 2811 leader_election.cc:290] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.487090 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f87840eaad4cf7a6f9c5101b114fbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250625 07:52:43.488420 986 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:43.489235 595 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet 656630686bfb49a59c5e7fc8a5cfb4b5
W20250625 07:52:43.490881 985 leader_election.cc:343] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 70094e6c7e644f62bf5ef04dcfb4c380
I20250625 07:52:43.490988 985 leader_election.cc:304] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:43.491328 2811 raft_consensus.cc:2747] T 70094e6c7e644f62bf5ef04dcfb4c380 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.495573 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.496632 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1498) took 3268 ms. Trace:
I20250625 07:52:43.496773 2407 rpcz_store.cc:276] 0625 07:52:40.227845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.227889 (+ 44us) service_pool.cc:225] Handling call
0625 07:52:43.047925 (+2820036us) ts_tablet_manager.cc:1145] Deleting tablet 065822c50586488785f7d9d53841a4dd
0625 07:52:43.047937 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.059454 (+ 11517us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.475716 (+416262us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.496605 (+ 20889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2819890}
I20250625 07:52:43.497224 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 065822c50586488785f7d9d53841a4dd (table default.Table_aada5cc96edc434b8e666d538d4eb5e8 [id=5c07ffb9bff2493da805be5a8cc4001e]) successfully deleted
I20250625 07:52:43.499125 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1499) took 3270 ms. Trace:
I20250625 07:52:43.499207 2407 rpcz_store.cc:276] 0625 07:52:40.228775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.228827 (+ 52us) service_pool.cc:225] Handling call
0625 07:52:43.497205 (+3268378us) ts_tablet_manager.cc:1145] Deleting tablet 71c66d9107024afa8b68deeac22332f6
0625 07:52:43.498946 (+ 1741us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.499108 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3268200}
I20250625 07:52:43.499591 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1500) took 3250 ms. Trace:
I20250625 07:52:43.499653 2407 rpcz_store.cc:276] 0625 07:52:40.249638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.250002 (+ 364us) service_pool.cc:225] Handling call
0625 07:52:43.499502 (+3249500us) ts_tablet_manager.cc:1145] Deleting tablet 99acd74c01f34d068c868fde3c6ec1df
0625 07:52:43.499510 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.499579 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249292}
I20250625 07:52:43.500033 1162 tablet_bootstrap.cc:492] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:43.500144 1162 ts_tablet_manager.cc:1397] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250625 07:52:43.500505 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:43.501816 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 71c66d9107024afa8b68deeac22332f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71c66d9107024afa8b68deeac22332f6 already in progress: creating tablet
W20250625 07:52:43.502123 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 99acd74c01f34d068c868fde3c6ec1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99acd74c01f34d068c868fde3c6ec1df already in progress: creating tablet
I20250625 07:52:43.501775 2407 tablet_replica.cc:331] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.503634 2407 raft_consensus.cc:2241] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.503820 2407 raft_consensus.cc:2270] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.504365 2407 ts_tablet_manager.cc:1905] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.505972 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 26d2bb8b0a8f45079083f2b42bc25011 (table default.Table_8adb7a38211942288ca93280cd05d4f5 [id=cc69077be31f4e739001440cf5bd3e67]) successfully deleted
I20250625 07:52:43.507738 2877 raft_consensus.cc:491] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.507951 2877 raft_consensus.cc:513] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.508013 1162 raft_consensus.cc:357] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.508426 1162 raft_consensus.cc:383] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.508493 1162 raft_consensus.cc:738] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.508939 2877 leader_election.cc:290] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.509384 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b5bc347c278c44c69444caf9ded72e1a" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.509898 713 leader_election.cc:343] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.515148 2723 raft_consensus.cc:491] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.515286 2723 raft_consensus.cc:513] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.515865 2723 leader_election.cc:290] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:43.516546 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfb421ea711a461890e267153250acea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.508725 1162 consensus_queue.cc:260] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.506592 1161 tablet_replica.cc:331] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: stopping tablet replica
I20250625 07:52:43.518883 1161 raft_consensus.cc:2241] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.519111 1161 raft_consensus.cc:2270] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.519706 911 raft_consensus.cc:1273] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092 [term 1 FOLLOWER]: Refusing update from remote peer 191b611b772a4fda9d1cba98776138a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:52:43.520013 1161 ts_tablet_manager.cc:1905] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 07:52:43.520256 851 leader_election.cc:343] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.524026 2869 raft_consensus.cc:491] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.524153 2869 raft_consensus.cc:513] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.524646 2869 leader_election.cc:290] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.525825 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9578d77db00242d48f4e8f6f8263ef09" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.526328 713 leader_election.cc:343] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.526580 1162 ts_tablet_manager.cc:1428] T 7e46f27344b848318dcd1e782819b75e P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250625 07:52:43.526901 1162 tablet_bootstrap.cc:492] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:43.537245 1162 tablet_bootstrap.cc:654] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:43.538033 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61ae72c767d14c3c8102b6be8a039690 consensus state has no leader
I20250625 07:52:43.542269 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_d4216890b0d74db484d0063a4caa6ac8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.542917 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4216890b0d74db484d0063a4caa6ac8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.543243 588 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_50038de54e04406b98642814cc8bded8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.543654 588 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50038de54e04406b98642814cc8bded8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.543969 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_2cb62c96e3c8496aba7760a59a7824a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.544327 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cb62c96e3c8496aba7760a59a7824a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.545289 2407 ts_tablet_manager.cc:1918] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.545405 2407 log.cc:1199] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/e10149f4ae0645ba88508ca51ae9f77a
I20250625 07:52:43.545694 2407 ts_tablet_manager.cc:1939] T e10149f4ae0645ba88508ca51ae9f77a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:43.546415 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1501) took 3296 ms. Trace:
I20250625 07:52:43.546494 2407 rpcz_store.cc:276] 0625 07:52:40.250385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.250431 (+ 46us) service_pool.cc:225] Handling call
0625 07:52:43.501734 (+3251303us) ts_tablet_manager.cc:1145] Deleting tablet e10149f4ae0645ba88508ca51ae9f77a
0625 07:52:43.501742 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.530736 (+ 28994us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.545164 (+ 14428us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.546395 (+ 1231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3251155}
I20250625 07:52:43.546845 2407 tablet_replica.cc:331] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.547029 2407 raft_consensus.cc:2241] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.547171 2407 raft_consensus.cc:2270] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.547706 2407 ts_tablet_manager.cc:1905] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.549007 2869 raft_consensus.cc:491] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.549137 2869 raft_consensus.cc:513] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.549717 2869 leader_election.cc:290] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.550707 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_9a51b748961b4e90971558af21f7b836" } new_table_name: "default.Table_e7eefabb57024d2886a98d6dd8d0130b" modify_external_catalogs: true
I20250625 07:52:43.563361 592 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
I20250625 07:52:43.570726 2723 raft_consensus.cc:491] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.570935 2723 raft_consensus.cc:513] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.571696 2723 leader_election.cc:290] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:43.572520 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8b5bb3fe3ef4bfdba93b862784a9f61" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.573029 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51c584e6bffb4f0a8020dadf78d718a7" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.573171 851 leader_election.cc:343] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:43.577870 713 leader_election.cc:343] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.570387 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_592077ca590f4f30adad6e11140574cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.579782 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_592077ca590f4f30adad6e11140574cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.581924 589 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet ffd98b2877b944f0bd484efa2a38a00a (table default.Table_e9dd87383290477db1eb357d644d1b82 [id=74ad120db7074f45acf606261432f7e8]). Expected version 1 got 0
I20250625 07:52:43.582139 589 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 1796a3305cb94ccb9a0596643a736a96 (table default.Table_0eb479026e15483586bb2df5333414d1 [id=5cb03ca1f790465aba5b692876e4f6d9]). Expected version 1 got 0
I20250625 07:52:43.583392 589 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_cf71021f057b438fbcb24d9ab68025a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.583871 589 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf71021f057b438fbcb24d9ab68025a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.589011 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 2bcfe7c3c54741fbb0aca69650fc8eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:17 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.591830 595 catalog_manager.cc:5674] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
I20250625 07:52:43.596499 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet e10149f4ae0645ba88508ca51ae9f77a (table default.Table_8c340466a42e47dead226f74c8176f49 [id=b086d131091348dcbd465e24e269721e]) successfully deleted
I20250625 07:52:43.596983 592 master_service.cc:948] ReplaceTablet: received request to replace tablet 883c5592cc014b8eb7d85f62242e5b02 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:43.600680 593 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_dab6e9af194049cb8259ae333b704be3" } new_table_name: "default.Table_4a33c509bed34f2fafbcf7174bd49956" modify_external_catalogs: true
W20250625 07:52:43.600785 590 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e538d6dd213f4b66870d314dadbf5c9a consensus state has no leader
I20250625 07:52:43.600984 596 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_debbb99725f945319fba9bd9aa86db4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.601171 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_57ea142c7cbe4cc183e919cfe37fb6c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.601478 596 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_debbb99725f945319fba9bd9aa86db4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:43.601691 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57ea142c7cbe4cc183e919cfe37fb6c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:43.602314 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fca218ee8594edeb0eaed16c81890dd consensus state has no leader
I20250625 07:52:43.602599 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_88c489ee92a349e69319d410220e7110" } modify_external_catalogs: true
I20250625 07:52:43.602741 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_88c489ee92a349e69319d410220e7110" } modify_external_catalogs: true
I20250625 07:52:43.603271 594 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_bba9d21266f54573897e4dd408928ab4" } new_table_name: "default.Table_a9551d83fd6c42cf9839832bf217d3d8" modify_external_catalogs: true
I20250625 07:52:43.607880 2723 raft_consensus.cc:491] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.608052 2723 raft_consensus.cc:513] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.608641 2723 leader_election.cc:290] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.609506 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef49f9f430824cf68fdd47e641039762" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
W20250625 07:52:43.610580 851 leader_election.cc:343] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.611565 592 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d85c53889892469d956fb4d28e74e2b0" } modify_external_catalogs: true
I20250625 07:52:43.611883 592 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_d85c53889892469d956fb4d28e74e2b0" } modify_external_catalogs: true
I20250625 07:52:43.613315 588 master_service.cc:948] ReplaceTablet: received request to replace tablet 52633d8b1d6e48b08ff52263c8a19496 from {username='slave'} at 127.0.0.1:34788
I20250625 07:52:43.617275 591 catalog_manager.cc:5674] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221) does not have the latest schema for tablet 61e076870efc4744b1224ddcb0eef3fa (table default.Table_3e4796a533d44263a8627ae547f7c8ba [id=9a2fe4250951446188a9e591f8f38b16]). Expected version 3 got 0
I20250625 07:52:43.619545 1012 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f47caf3bae41b79b0aa1e320f15da4 with delete_type TABLET_DATA_DELETED (replaced by tablet cac7d20327e14f90af25acb58d724b68) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.619756 1011 tablet_service.cc:1515] Processing DeleteTablet for tablet 70094e6c7e644f62bf5ef04dcfb4c380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:24 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.619877 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet fe630469fe7c4f4bac28277896eddc9c with delete_type TABLET_DATA_DELETED (replaced by tablet f815ed67e17347dcad804cbc1ba7660c) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.624519 2407 ts_tablet_manager.cc:1918] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.625591 2407 log.cc:1199] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/034903d699584feb91f35cbb2f62701a
I20250625 07:52:43.629276 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8b5bb3fe3ef4bfdba93b862784a9f61" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.629359 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfb421ea711a461890e267153250acea" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.629722 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef49f9f430824cf68fdd47e641039762" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.629796 595 catalog_manager.cc:5674] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661) does not have the latest schema for tablet 34544157495f4e14a7978e0eec70b522 (table default.Table_bb53a82b1c824335b04789e50a88c91f [id=04e37ec20f144d578c85368b35713aa9]). Expected version 1 got 0
W20250625 07:52:43.629940 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet a571983779854eb398b3b0b42c34f83a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a571983779854eb398b3b0b42c34f83a already in progress: creating tablet
I20250625 07:52:43.630009 595 catalog_manager.cc:5582] T e35d77471f7d4866a8c30324b5da0170 P 191b611b772a4fda9d1cba98776138a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 191b611b772a4fda9d1cba98776138a7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "191b611b772a4fda9d1cba98776138a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } health_report { overall_health: UNKNOWN } } }
I20250625 07:52:43.631147 1160 ts_tablet_manager.cc:1918] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.631186 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 656630686bfb49a59c5e7fc8a5cfb4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.631245 1160 log.cc:1199] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/0102a3b176e146a8b19abe22d3aba656
I20250625 07:52:43.631361 2407 ts_tablet_manager.cc:1939] T 034903d699584feb91f35cbb2f62701a P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
W20250625 07:52:43.635718 593 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48573933194547a0810fd1742294d525 consensus state has no leader
I20250625 07:52:43.636047 880 tablet_service.cc:1515] Processing DeleteTablet for tablet c4d3bff930cf4ce7941d2a9608f889b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.636240 878 tablet_service.cc:1515] Processing DeleteTablet for tablet bfb421ea711a461890e267153250acea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:23 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.636272 1160 ts_tablet_manager.cc:1939] T 0102a3b176e146a8b19abe22d3aba656 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
W20250625 07:52:43.628600 850 leader_election.cc:343] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.638538 850 leader_election.cc:304] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.638727 1163 raft_consensus.cc:357] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.638921 1163 raft_consensus.cc:383] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.638974 1163 raft_consensus.cc:738] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.639156 1163 consensus_queue.cc:260] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
W20250625 07:52:43.639220 850 leader_election.cc:343] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.639321 850 leader_election.cc:304] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.639536 2861 raft_consensus.cc:2747] T 5b19f6c181ee4e24883b0d7df822ab3a P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.639691 1163 ts_tablet_manager.cc:1428] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.229s user 0.002s sys 0.000s
I20250625 07:52:43.639808 2861 raft_consensus.cc:2747] T b8b5bb3fe3ef4bfdba93b862784a9f61 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.640353 850 leader_election.cc:343] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.640455 850 leader_election.cc:304] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.639505 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1502) took 3388 ms. Trace:
I20250625 07:52:43.640640 2861 raft_consensus.cc:2747] T bfb421ea711a461890e267153250acea P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.640976 850 leader_election.cc:343] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.641069 850 leader_election.cc:304] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.641407 2861 raft_consensus.cc:2747] T ef49f9f430824cf68fdd47e641039762 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.640843 2407 rpcz_store.cc:276] 0625 07:52:40.250790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.250831 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:43.546799 (+3295968us) ts_tablet_manager.cc:1145] Deleting tablet 034903d699584feb91f35cbb2f62701a
0625 07:52:43.546808 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.596045 (+ 49237us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.623741 (+ 27696us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.638842 (+ 15101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3295822}
I20250625 07:52:43.642252 2407 tablet_replica.cc:331] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.643024 2407 raft_consensus.cc:2241] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.643672 2407 raft_consensus.cc:2270] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.644392 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet eba09cd15d9f4611ab14d4a6dc372719
I20250625 07:52:43.645484 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
W20250625 07:52:43.647030 713 leader_election.cc:343] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: d0a75906385744e5ba6938f451a2d7ec
I20250625 07:52:43.653033 713 leader_election.cc:304] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.653576 2869 raft_consensus.cc:2747] T d0a75906385744e5ba6938f451a2d7ec P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.654913 594 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a72e472025348da8510f66904424a2b consensus state has no leader
I20250625 07:52:43.655532 2869 consensus_queue.cc:1035] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:43.655974 594 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a" } modify_external_catalogs: true
I20250625 07:52:43.656141 594 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_8b0d01da1bb144aaa38bdd0b63b24a3a" } modify_external_catalogs: true
I20250625 07:52:43.656455 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1474) took 2690 ms. Trace:
I20250625 07:52:43.656539 1160 rpcz_store.cc:276] 0625 07:52:40.966177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.000259 (+ 34082us) service_pool.cc:225] Handling call
0625 07:52:43.444920 (+2444661us) ts_tablet_manager.cc:1145] Deleting tablet 0102a3b176e146a8b19abe22d3aba656
0625 07:52:43.444931 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.489539 (+ 44608us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.631053 (+141514us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.656433 (+ 25380us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19221,"tablet-delete.queue_time_us":2442454}
I20250625 07:52:43.656894 1160 tablet_replica.cc:331] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:43.657084 1160 raft_consensus.cc:2241] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.657232 1160 raft_consensus.cc:2270] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.657743 1160 ts_tablet_manager.cc:1905] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.659188 1163 tablet_bootstrap.cc:492] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:43.659459 878 tablet_service.cc:1515] Processing DeleteTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.661273 1163 tablet_bootstrap.cc:654] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.663625 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 0102a3b176e146a8b19abe22d3aba656 (table default.Table_f4184bdee984426ea3612ca34ca1bf92 [id=2a743c4696284539941ed3c1309a1660]) successfully deleted
I20250625 07:52:43.668769 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 034903d699584feb91f35cbb2f62701a (table default.Table_bc60ae6057e24cb8aa07fc94a678c0d9 [id=17f7f02b235944ffa0fdbe5c1af3492d]) successfully deleted
I20250625 07:52:43.673811 2723 raft_consensus.cc:491] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.673981 2723 raft_consensus.cc:513] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.674593 2723 leader_election.cc:290] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661)
I20250625 07:52:43.674789 597 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b374bdf422d9423b85441edf89cc3c95" } modify_external_catalogs: true
I20250625 07:52:43.675390 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a80d81cc2d524f318b858a9bd97a89b8" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.675464 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a80d81cc2d524f318b858a9bd97a89b8" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.675972 850 leader_election.cc:343] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:43.676146 851 leader_election.cc:343] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.676281 851 leader_election.cc:304] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.676702 2723 raft_consensus.cc:2747] T a80d81cc2d524f318b858a9bd97a89b8 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.677095 597 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_b374bdf422d9423b85441edf89cc3c95" } modify_external_catalogs: true
I20250625 07:52:43.683207 1160 ts_tablet_manager.cc:1918] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.683409 1160 log.cc:1199] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/7e46f27344b848318dcd1e782819b75e
I20250625 07:52:43.683804 1160 ts_tablet_manager.cc:1939] T 7e46f27344b848318dcd1e782819b75e P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:43.684643 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1475) took 2718 ms. Trace:
W20250625 07:52:43.684929 713 leader_election.cc:343] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: b5bc347c278c44c69444caf9ded72e1a
I20250625 07:52:43.685099 713 leader_election.cc:304] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.685376 2869 raft_consensus.cc:2747] T b5bc347c278c44c69444caf9ded72e1a P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.686019 713 leader_election.cc:343] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 51c584e6bffb4f0a8020dadf78d718a7
I20250625 07:52:43.686157 713 leader_election.cc:304] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.686394 2877 raft_consensus.cc:2747] T 51c584e6bffb4f0a8020dadf78d718a7 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.687198 713 leader_election.cc:343] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 9578d77db00242d48f4e8f6f8263ef09
I20250625 07:52:43.687356 713 leader_election.cc:304] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.687610 2877 raft_consensus.cc:2747] T 9578d77db00242d48f4e8f6f8263ef09 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.690995 755 tablet_service.cc:1515] Processing DeleteTablet for tablet eba09cd15d9f4611ab14d4a6dc372719 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.691270 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e46f27344b848318dcd1e782819b75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:26 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.692938 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 47c9aca996884ffeab88381dd3cb189b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:28 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.694306 589 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca9bb00cf9834f0797220f17c7ad930d" } modify_external_catalogs: true
I20250625 07:52:43.694483 589 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ca9bb00cf9834f0797220f17c7ad930d" } modify_external_catalogs: true
I20250625 07:52:43.696071 594 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet de3b07b2e0284761a96ea3650506638b
I20250625 07:52:43.697234 2407 ts_tablet_manager.cc:1905] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.700942 597 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet e92dc108e9fb4942a51cec78da1f0930
I20250625 07:52:43.702001 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e92dc108e9fb4942a51cec78da1f0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.714062 2877 consensus_queue.cc:1035] T ea2b581d722d4a8283005b37106f5b51 P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:52:43.715001 747 tablet_service.cc:1515] Processing DeleteTablet for tablet de3b07b2e0284761a96ea3650506638b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.715221 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e92dc108e9fb4942a51cec78da1f0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.721275 2877 raft_consensus.cc:491] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.721465 2877 raft_consensus.cc:513] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.722138 2877 leader_election.cc:290] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.728477 589 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 3eb542ac24f6405687c1067f2885bec6: Sending DeleteTablet for 3 replicas of tablet fb22dd02b6d44396a13710f6463f1b93
I20250625 07:52:43.732101 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet de3b07b2e0284761a96ea3650506638b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.700310 878 tablet_service.cc:1515] Processing DeleteTablet for tablet de3b07b2e0284761a96ea3650506638b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.701649 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e92dc108e9fb4942a51cec78da1f0930 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
I20250625 07:52:43.732645 1160 rpcz_store.cc:276] 0625 07:52:40.966248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.002493 (+ 36245us) service_pool.cc:225] Handling call
0625 07:52:43.656839 (+2654346us) ts_tablet_manager.cc:1145] Deleting tablet 7e46f27344b848318dcd1e782819b75e
0625 07:52:43.656852 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.665789 (+ 8937us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.683083 (+ 17294us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.684621 (+ 1538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654198}
I20250625 07:52:43.732860 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet fb22dd02b6d44396a13710f6463f1b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.733114 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1476) took 2766 ms. Trace:
I20250625 07:52:43.733100 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 7e46f27344b848318dcd1e782819b75e (table default.Table_9bc3d032aee948de91f3c5c8c776cf7d [id=4fff90a74052472bb7bb8778a9786b02]) successfully deleted
I20250625 07:52:43.733182 1160 rpcz_store.cc:276] 0625 07:52:40.966341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.002648 (+ 36307us) service_pool.cc:225] Handling call
0625 07:52:43.732975 (+2730327us) ts_tablet_manager.cc:1145] Deleting tablet fd6c3a490883477dbafcb74d39be7345
0625 07:52:43.732986 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.733097 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730204}
I20250625 07:52:43.733275 2811 raft_consensus.cc:491] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.733407 2811 raft_consensus.cc:513] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:43.733500 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet fd6c3a490883477dbafcb74d39be7345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd6c3a490883477dbafcb74d39be7345 already in progress: creating tablet
I20250625 07:52:43.733551 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1477) took 2767 ms. Trace:
I20250625 07:52:43.713866 1163 tablet_bootstrap.cc:492] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:43.733616 1160 rpcz_store.cc:276] 0625 07:52:40.966404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.002775 (+ 36371us) service_pool.cc:225] Handling call
0625 07:52:43.733380 (+2730605us) ts_tablet_manager.cc:1145] Deleting tablet e15c7ac8ff934a8ca40ac480c7c352ab
0625 07:52:43.733388 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.733534 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730506}
W20250625 07:52:43.733943 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e15c7ac8ff934a8ca40ac480c7c352ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e15c7ac8ff934a8ca40ac480c7c352ab already in progress: creating tablet
I20250625 07:52:43.734090 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1478) took 2682 ms. Trace:
I20250625 07:52:43.734097 2811 leader_election.cc:290] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.734189 1163 ts_tablet_manager.cc:1397] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.002s
I20250625 07:52:43.730046 893 tablet_service.cc:1515] Processing DeleteTablet for tablet fb22dd02b6d44396a13710f6463f1b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:34046
W20250625 07:52:43.735435 986 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 84f47caf3bae41b79b0aa1e320f15da4
I20250625 07:52:43.736152 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "506794253c84486faa926de885c48c58" candidate_uuid: "191b611b772a4fda9d1cba98776138a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
I20250625 07:52:43.737141 1163 raft_consensus.cc:357] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.737319 1163 raft_consensus.cc:383] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.737380 1163 raft_consensus.cc:738] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.737578 1163 consensus_queue.cc:260] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.738169 1163 ts_tablet_manager.cc:1428] T f8fe5a9156b14d1ea79fd1f3a04dcd20 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250625 07:52:43.738435 1163 tablet_bootstrap.cc:492] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:43.740049 1163 tablet_bootstrap.cc:654] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.740211 597 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_3e7b17a7d5fc49d59f569ce163401177" } new_table_name: "default.Table_32b69a40406146d292d4f99835659963" modify_external_catalogs: true
I20250625 07:52:43.741412 2723 raft_consensus.cc:491] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:43.741514 985 leader_election.cc:343] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 84f47caf3bae41b79b0aa1e320f15da4
I20250625 07:52:43.741545 2723 raft_consensus.cc:513] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.741617 985 leader_election.cc:304] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:43.742005 2811 raft_consensus.cc:2747] T 84f47caf3bae41b79b0aa1e320f15da4 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.741943 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_9306129152d44ae6b4fa2e2f7c253764"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.742071 2723 leader_election.cc:290] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221)
I20250625 07:52:43.742643 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a29a9ba27ab4eb494e1b91e4b33aa75" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "191b611b772a4fda9d1cba98776138a7" is_pre_election: true
I20250625 07:52:43.742759 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a29a9ba27ab4eb494e1b91e4b33aa75" candidate_uuid: "429026e8d98d46339db143f84897a092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" is_pre_election: true
W20250625 07:52:43.742663 713 leader_election.cc:343] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 506794253c84486faa926de885c48c58
W20250625 07:52:43.743405 713 leader_election.cc:343] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.743506 713 leader_election.cc:304] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 191b611b772a4fda9d1cba98776138a7; no voters: 429026e8d98d46339db143f84897a092, 8f2cdaf87b9d4b2a919098bf65efd513
W20250625 07:52:43.743741 850 leader_election.cc:343] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:52:43.744113 582 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4fb211d352ae4ee6b433376787420700 (table default.Table_edd9a0a8a5ec47e6bdfb9264fdbc961d [id=97abf3cac6584696ba07c6db0822b5ae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4fb211d352ae4ee6b433376787420700 consensus state has no leader
W20250625 07:52:43.744132 851 leader_election.cc:343] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:52:43.744216 851 leader_election.cc:304] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429026e8d98d46339db143f84897a092; no voters: 191b611b772a4fda9d1cba98776138a7, 8f2cdaf87b9d4b2a919098bf65efd513
I20250625 07:52:43.744531 2723 raft_consensus.cc:2747] T 3a29a9ba27ab4eb494e1b91e4b33aa75 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.744923 743 tablet_service.cc:1515] Processing DeleteTablet for tablet fb22dd02b6d44396a13710f6463f1b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:43 UTC) from {username='slave'} at 127.0.0.1:33894
I20250625 07:52:43.748955 2876 raft_consensus.cc:2747] T 506794253c84486faa926de885c48c58 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:52:43.730578 2894 tablet.cc:1722] T 3823279d7e06435484f1b82836ebcf39 P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250625 07:52:43.751533 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9306129152d44ae6b4fa2e2f7c253764 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.753679 1162 tablet_bootstrap.cc:492] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:43.753857 1162 ts_tablet_manager.cc:1397] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.227s user 0.009s sys 0.015s
I20250625 07:52:43.754899 1162 raft_consensus.cc:357] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
W20250625 07:52:43.757324 597 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3522ef3ed0714b60944a851bad7375fc consensus state has no leader
I20250625 07:52:43.734166 1160 rpcz_store.cc:276] 0625 07:52:41.051719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.051779 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:43.733959 (+2682180us) ts_tablet_manager.cc:1145] Deleting tablet d43d98c0c51b40c6bdb0c2705465038e
0625 07:52:43.733967 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.734075 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681984}
I20250625 07:52:43.762197 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1479) took 2710 ms. Trace:
I20250625 07:52:43.762540 2662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250625 07:52:43.764343 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet d43d98c0c51b40c6bdb0c2705465038e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d43d98c0c51b40c6bdb0c2705465038e already in progress: creating tablet
I20250625 07:52:43.765089 1160 rpcz_store.cc:276] 0625 07:52:41.052132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.052179 (+ 47us) service_pool.cc:225] Handling call
0625 07:52:43.762061 (+2709882us) ts_tablet_manager.cc:1145] Deleting tablet 670974e43d7e4886bfe5eb3cf36ab6ce
0625 07:52:43.762071 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.762177 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709739}
I20250625 07:52:43.765563 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1480) took 2676 ms. Trace:
I20250625 07:52:43.765668 1160 rpcz_store.cc:276] 0625 07:52:41.088922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.088975 (+ 53us) service_pool.cc:225] Handling call
0625 07:52:43.765426 (+2676451us) ts_tablet_manager.cc:1145] Deleting tablet 9b83f4c8544946d9adb3346df200b215
0625 07:52:43.765439 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.765544 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676248}
I20250625 07:52:43.766005 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1481) took 2521 ms. Trace:
I20250625 07:52:43.766089 1160 rpcz_store.cc:276] 0625 07:52:41.244635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.245044 (+ 409us) service_pool.cc:225] Handling call
0625 07:52:43.765894 (+2520850us) ts_tablet_manager.cc:1145] Deleting tablet f2909c29db4c434c9db393c12ea48a1d
0625 07:52:43.765903 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.765988 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2520602}
I20250625 07:52:43.766413 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1483) took 2437 ms. Trace:
I20250625 07:52:43.766494 1160 rpcz_store.cc:276] 0625 07:52:41.329264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.329659 (+ 395us) service_pool.cc:225] Handling call
0625 07:52:43.766309 (+2436650us) ts_tablet_manager.cc:1145] Deleting tablet dba8780a44864169ac9cd63ab2e89fe9
0625 07:52:43.766320 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.766398 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436436}
I20250625 07:52:43.766855 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1482) took 2437 ms. Trace:
I20250625 07:52:43.766937 1160 rpcz_store.cc:276] 0625 07:52:41.329001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.329060 (+ 59us) service_pool.cc:225] Handling call
0625 07:52:43.766742 (+2437682us) ts_tablet_manager.cc:1145] Deleting tablet b3a84f7d816148788725216f713b3980
0625 07:52:43.766754 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.766838 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436703}
I20250625 07:52:43.767259 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1484) took 2378 ms. Trace:
I20250625 07:52:43.767338 1160 rpcz_store.cc:276] 0625 07:52:41.388528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.398102 (+ 9574us) service_pool.cc:225] Handling call
0625 07:52:43.767154 (+2369052us) ts_tablet_manager.cc:1145] Deleting tablet e609e7267a66453dab2b666d69aebd02
0625 07:52:43.767166 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.767243 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368089}
I20250625 07:52:43.767604 1160 tablet_replica.cc:331] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: stopping tablet replica
I20250625 07:52:43.767879 1160 raft_consensus.cc:2241] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.768056 1160 raft_consensus.cc:2270] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250625 07:52:43.769143 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 670974e43d7e4886bfe5eb3cf36ab6ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670974e43d7e4886bfe5eb3cf36ab6ce already in progress: creating tablet
W20250625 07:52:43.769381 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 9b83f4c8544946d9adb3346df200b215 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b83f4c8544946d9adb3346df200b215 already in progress: creating tablet
W20250625 07:52:43.769562 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet f2909c29db4c434c9db393c12ea48a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2909c29db4c434c9db393c12ea48a1d already in progress: creating tablet
W20250625 07:52:43.769744 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet dba8780a44864169ac9cd63ab2e89fe9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dba8780a44864169ac9cd63ab2e89fe9 already in progress: creating tablet
W20250625 07:52:43.769930 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet b3a84f7d816148788725216f713b3980 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3a84f7d816148788725216f713b3980 already in progress: creating tablet
W20250625 07:52:43.770113 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet e609e7267a66453dab2b666d69aebd02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e609e7267a66453dab2b666d69aebd02 already in progress: creating tablet
I20250625 07:52:43.775545 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_4b0bc9b7a2804f4f97ce0aeac79c9463"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.776141 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b0bc9b7a2804f4f97ce0aeac79c9463 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.755143 1162 raft_consensus.cc:383] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.777276 1162 raft_consensus.cc:738] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.777480 1162 consensus_queue.cc:260] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.778038 1162 ts_tablet_manager.cc:1428] T 65231ee1a59d49eb96ba352b74c3460c P 191b611b772a4fda9d1cba98776138a7: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250625 07:52:43.755975 2892 tablet.cc:1722] T 3823279d7e06435484f1b82836ebcf39 P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250625 07:52:43.780386 1161 ts_tablet_manager.cc:1918] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.776530 2407 ts_tablet_manager.cc:1918] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.772078 2876 consensus_queue.cc:1035] T 62aaff3c000e4e69b52222aa47260c3a P 191b611b772a4fda9d1cba98776138a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250625 07:52:43.782567 2407 log.cc:1199] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/08cd482ceec244c2bed2b116d5c14d89
I20250625 07:52:43.783066 2896 tablet.cc:1722] T d1c96cdddd3443b8b67f57f4d6c4846e P 191b611b772a4fda9d1cba98776138a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:43.783234 2407 ts_tablet_manager.cc:1939] T 08cd482ceec244c2bed2b116d5c14d89 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:43.784638 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1503) took 3533 ms. Trace:
I20250625 07:52:43.784759 2407 rpcz_store.cc:276] 0625 07:52:40.251105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.252447 (+ 1342us) service_pool.cc:225] Handling call
0625 07:52:43.642112 (+3389665us) ts_tablet_manager.cc:1145] Deleting tablet 08cd482ceec244c2bed2b116d5c14d89
0625 07:52:43.642123 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.701395 (+ 59272us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.776235 (+ 74840us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.784620 (+ 8385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3389485}
I20250625 07:52:43.785121 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1505) took 3531 ms. Trace:
I20250625 07:52:43.785202 2407 rpcz_store.cc:276] 0625 07:52:40.253279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.253320 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:43.785003 (+3531683us) ts_tablet_manager.cc:1145] Deleting tablet ba49ad0cb12d4f738288aea215dbda87
0625 07:52:43.785014 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.785106 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3531525}
I20250625 07:52:43.785495 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1506) took 3531 ms. Trace:
I20250625 07:52:43.785562 2407 rpcz_store.cc:276] 0625 07:52:40.253637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.253680 (+ 43us) service_pool.cc:225] Handling call
0625 07:52:43.785405 (+3531725us) ts_tablet_manager.cc:1145] Deleting tablet 7b3eafac1b2b46acb2ac6fca087b7f16
0625 07:52:43.785414 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.785483 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3531564}
I20250625 07:52:43.785846 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1507) took 3531 ms. Trace:
I20250625 07:52:43.785912 2407 rpcz_store.cc:276] 0625 07:52:40.254054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.254095 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:43.785752 (+3531657us) ts_tablet_manager.cc:1145] Deleting tablet b4d68f1d9d8244a2b49f5e1a5f2855f2
0625 07:52:43.785761 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.785833 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3531527}
I20250625 07:52:43.786149 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1508) took 3531 ms. Trace:
I20250625 07:52:43.786199 2407 rpcz_store.cc:276] 0625 07:52:40.254337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.254375 (+ 38us) service_pool.cc:225] Handling call
0625 07:52:43.786071 (+3531696us) ts_tablet_manager.cc:1145] Deleting tablet cd226579c77b4938bb031fac0412cbb1
0625 07:52:43.786078 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.786137 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3531570}
I20250625 07:52:43.786417 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1509) took 3531 ms. Trace:
I20250625 07:52:43.786408 1161 log.cc:1199] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-2/wal/wals/23af43b9ceeb4f2b812ee251ea8545ce
I20250625 07:52:43.786466 2407 rpcz_store.cc:276] 0625 07:52:40.254646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.254691 (+ 45us) service_pool.cc:225] Handling call
0625 07:52:43.786347 (+3531656us) ts_tablet_manager.cc:1145] Deleting tablet f64c42861e60451e92131a435f95b012
0625 07:52:43.786353 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.786406 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3531527}
I20250625 07:52:43.786717 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1512) took 3316 ms. Trace:
I20250625 07:52:43.786785 1161 ts_tablet_manager.cc:1939] T 23af43b9ceeb4f2b812ee251ea8545ce P 8f2cdaf87b9d4b2a919098bf65efd513: Deleting consensus metadata
I20250625 07:52:43.786770 2407 rpcz_store.cc:276] 0625 07:52:40.470491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.471157 (+ 666us) service_pool.cc:225] Handling call
0625 07:52:43.786608 (+3315451us) ts_tablet_manager.cc:1145] Deleting tablet d320734990404d1e9222bea1d88a18fe
0625 07:52:43.786649 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.786706 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3315211}
I20250625 07:52:43.787001 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1516) took 2883 ms. Trace:
I20250625 07:52:43.787056 2407 rpcz_store.cc:276] 0625 07:52:40.904001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.904042 (+ 41us) service_pool.cc:225] Handling call
0625 07:52:43.786931 (+2882889us) ts_tablet_manager.cc:1145] Deleting tablet 1bbf1cfac4e14a20b3fb53607cdf100f
0625 07:52:43.786937 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.786990 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2882703}
I20250625 07:52:43.787271 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1519) took 2870 ms. Trace:
I20250625 07:52:43.787320 2407 rpcz_store.cc:276] 0625 07:52:40.916934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:40.917000 (+ 66us) service_pool.cc:225] Handling call
0625 07:52:43.787203 (+2870203us) ts_tablet_manager.cc:1145] Deleting tablet 6d6b4bca32de4a538071a652409e93b2
0625 07:52:43.787210 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.787261 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2869979}
I20250625 07:52:43.787460 583 catalog_manager.cc:4928] TS 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221): tablet 23af43b9ceeb4f2b812ee251ea8545ce (table default.Table_cb6a73fc11c44f4ead125864cc91710d [id=9fc131343ad246b58a3bf2a42d29e4ef]) successfully deleted
I20250625 07:52:43.787528 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1550) took 2598 ms. Trace:
I20250625 07:52:43.787576 2407 rpcz_store.cc:276] 0625 07:52:41.189317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.199590 (+ 10273us) service_pool.cc:225] Handling call
0625 07:52:43.787460 (+2587870us) ts_tablet_manager.cc:1145] Deleting tablet a571983779854eb398b3b0b42c34f83a
0625 07:52:43.787466 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.787518 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587558}
I20250625 07:52:43.787765 2407 tablet_replica.cc:331] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.787919 2407 raft_consensus.cc:2241] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.788038 2407 raft_consensus.cc:2270] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.789196 2897 tablet.cc:1722] T d1c96cdddd3443b8b67f57f4d6c4846e P 429026e8d98d46339db143f84897a092: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250625 07:52:43.791476 584 catalog_manager.cc:4928] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): tablet 08cd482ceec244c2bed2b116d5c14d89 (table default.Table_21de3cd434744102b0d5ffbbc17c18c3 [id=f129efbf6ddb47d1a59e7f511637654a]) successfully deleted
W20250625 07:52:43.792014 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet ba49ad0cb12d4f738288aea215dbda87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba49ad0cb12d4f738288aea215dbda87 already in progress: creating tablet
I20250625 07:52:43.793119 2662 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:52:43.793550 2662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:52:43.795065 1162 tablet_bootstrap.cc:492] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7: Bootstrap starting.
I20250625 07:52:43.799165 2877 raft_consensus.cc:491] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.799316 2877 raft_consensus.cc:513] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.799930 2877 leader_election.cc:290] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.806815 1160 ts_tablet_manager.cc:1905] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.807225 1158 tablet_bootstrap.cc:492] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.807384 1158 ts_tablet_manager.cc:1397] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.339s user 0.025s sys 0.062s
I20250625 07:52:43.807695 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_2eb4cc88dc8c42eeb3bdee821d48710d" } new_table_name: "default.Table_7d270eca1930403590e13ae8e22f4f70" modify_external_catalogs: true
I20250625 07:52:43.808123 1158 raft_consensus.cc:357] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.808285 1158 raft_consensus.cc:383] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.808336 1158 raft_consensus.cc:738] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.808526 1158 consensus_queue.cc:260] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.809017 1158 ts_tablet_manager.cc:1428] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250625 07:52:43.811372 1163 tablet_bootstrap.cc:492] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:43.811486 1163 ts_tablet_manager.cc:1397] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.000s
I20250625 07:52:43.812181 1163 raft_consensus.cc:357] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.812326 1163 raft_consensus.cc:383] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.812367 1163 raft_consensus.cc:738] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.812525 1163 consensus_queue.cc:260] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.812942 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_7b1b7200f1774ebd955e7524c2c2119e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.813107 1163 ts_tablet_manager.cc:1428] T 73ec635d7d5242e8804dd3527457a540 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250625 07:52:43.813347 1163 tablet_bootstrap.cc:492] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
W20250625 07:52:43.813512 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b1b7200f1774ebd955e7524c2c2119e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.813796 2407 ts_tablet_manager.cc:1905] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.814970 1163 tablet_bootstrap.cc:654] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.816586 1158 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.816972 589 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_ae8736f8b7954519bedb73c2427040e6" } new_table_name: "default.Table_6a6dbaf42c294791a0b9d0768f09870e" modify_external_catalogs: true
I20250625 07:52:43.818425 1158 tablet_bootstrap.cc:654] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.819640 591 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_58ade13d3dc04e0ba19a0959923f8575"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.820145 591 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58ade13d3dc04e0ba19a0959923f8575 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.820451 1160 ts_tablet_manager.cc:1918] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.820544 1160 log.cc:1199] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/4f1c150fb54c405c8dc61ac133d98b13
I20250625 07:52:43.820852 1160 ts_tablet_manager.cc:1939] T 4f1c150fb54c405c8dc61ac133d98b13 P 429026e8d98d46339db143f84897a092: Deleting consensus metadata
I20250625 07:52:43.821588 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1485) took 2199 ms. Trace:
I20250625 07:52:43.821676 1160 rpcz_store.cc:276] 0625 07:52:41.621895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.621953 (+ 58us) service_pool.cc:225] Handling call
0625 07:52:43.767553 (+2145600us) ts_tablet_manager.cc:1145] Deleting tablet 4f1c150fb54c405c8dc61ac133d98b13
0625 07:52:43.767564 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.817495 (+ 49931us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.820363 (+ 2868us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.821569 (+ 1206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2145360}
I20250625 07:52:43.822724 590 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_eb3198b0194245b4b6ac8b21fa8bbf5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.823207 590 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb3198b0194245b4b6ac8b21fa8bbf5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.824297 594 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_de12df941dcb42ff87757ec0d56277c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.825508 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 3cdbdd1e3b734463ab452bafecb75822 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 07:52:31 UTC) from {username='slave'} at 127.0.0.1:47274
I20250625 07:52:43.827770 2407 ts_tablet_manager.cc:1918] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.827886 2407 log.cc:1199] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-0/wal/wals/1c8694f09dcf47ab9f5fc8854c21c331
I20250625 07:52:43.828161 2407 ts_tablet_manager.cc:1939] T 1c8694f09dcf47ab9f5fc8854c21c331 P 191b611b772a4fda9d1cba98776138a7: Deleting consensus metadata
I20250625 07:52:43.829309 583 catalog_manager.cc:4928] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): tablet 4f1c150fb54c405c8dc61ac133d98b13 (table default.Table_b36659e6e06d41298f3a614061f97b73 [id=76b5f05d74fa415a93eb6df1c6da6a8e]) successfully deleted
I20250625 07:52:43.830660 2811 raft_consensus.cc:491] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250625 07:52:43.830708 594 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de12df941dcb42ff87757ec0d56277c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.830811 2811 raft_consensus.cc:513] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.831419 2811 leader_election.cc:290] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.831413 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1489) took 2151 ms. Trace:
I20250625 07:52:43.831496 1160 rpcz_store.cc:276] 0625 07:52:41.679865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.679929 (+ 64us) service_pool.cc:225] Handling call
0625 07:52:43.821959 (+2142030us) ts_tablet_manager.cc:1145] Deleting tablet acfea07d8441415e92d59f2f4e0b72a6
0625 07:52:43.821972 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.831390 (+ 9418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2141823}
I20250625 07:52:43.832113 597 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_e0e5746bd5ea44cfb725edd67ec8abc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:52:43.832836 986 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): Not found: Tablet not found: 3cdbdd1e3b734463ab452bafecb75822
W20250625 07:52:43.833140 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet acfea07d8441415e92d59f2f4e0b72a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet acfea07d8441415e92d59f2f4e0b72a6 already in progress: creating tablet
W20250625 07:52:43.833314 985 leader_election.cc:343] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): Not found: Tablet not found: 3cdbdd1e3b734463ab452bafecb75822
I20250625 07:52:43.833410 985 leader_election.cc:304] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f2cdaf87b9d4b2a919098bf65efd513; no voters: 191b611b772a4fda9d1cba98776138a7, 429026e8d98d46339db143f84897a092
I20250625 07:52:43.833482 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1490) took 2151 ms. Trace:
I20250625 07:52:43.833547 1160 rpcz_store.cc:276] 0625 07:52:41.681709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.681806 (+ 97us) service_pool.cc:225] Handling call
0625 07:52:43.833387 (+2151581us) ts_tablet_manager.cc:1145] Deleting tablet 4ee3563a2d994e6e88d2e0715a7b3816
0625 07:52:43.833396 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.833468 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2151403}
I20250625 07:52:43.833901 2811 raft_consensus.cc:2747] T 3cdbdd1e3b734463ab452bafecb75822 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250625 07:52:43.833985 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 4ee3563a2d994e6e88d2e0715a7b3816 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ee3563a2d994e6e88d2e0715a7b3816 already in progress: creating tablet
I20250625 07:52:43.834498 2407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33894 (request call id 1553) took 2644 ms. Trace:
I20250625 07:52:43.834599 2407 rpcz_store.cc:276] 0625 07:52:41.189541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.202227 (+ 12686us) service_pool.cc:225] Handling call
0625 07:52:43.787732 (+2585505us) ts_tablet_manager.cc:1145] Deleting tablet 1c8694f09dcf47ab9f5fc8854c21c331
0625 07:52:43.787739 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.824821 (+ 37082us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.827689 (+ 2868us) tablet_metadata.cc:641] Metadata flushed
0625 07:52:43.834479 (+ 6790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2585322}
I20250625 07:52:43.834863 2407 tablet_replica.cc:331] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: stopping tablet replica
I20250625 07:52:43.835032 2407 raft_consensus.cc:2241] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.835186 2407 raft_consensus.cc:2270] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.835650 2407 ts_tablet_manager.cc:1905] T a396bde4b5964697ad0090eab777ba0e P 191b611b772a4fda9d1cba98776138a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.836833 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1491) took 2150 ms. Trace:
I20250625 07:52:43.836956 1160 rpcz_store.cc:276] 0625 07:52:41.686894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.687199 (+ 305us) service_pool.cc:225] Handling call
0625 07:52:43.836736 (+2149537us) ts_tablet_manager.cc:1145] Deleting tablet 565d312783f24b9b995595f413ce4697
0625 07:52:43.836745 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.836819 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149311}
W20250625 07:52:43.837739 597 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e5746bd5ea44cfb725edd67ec8abc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250625 07:52:43.838173 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 565d312783f24b9b995595f413ce4697 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565d312783f24b9b995595f413ce4697 already in progress: creating tablet
I20250625 07:52:43.841926 1163 tablet_bootstrap.cc:492] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092: No bootstrap required, opened a new log
I20250625 07:52:43.842114 1163 ts_tablet_manager.cc:1397] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.002s
I20250625 07:52:43.844551 1163 raft_consensus.cc:357] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.845494 1163 raft_consensus.cc:383] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.845646 1163 raft_consensus.cc:738] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429026e8d98d46339db143f84897a092, State: Initialized, Role: FOLLOWER
W20250625 07:52:43.846066 596 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f863462f3684d249a732033940b489b consensus state has no leader
I20250625 07:52:43.846604 1158 tablet_bootstrap.cc:492] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513: No bootstrap required, opened a new log
I20250625 07:52:43.846843 1158 ts_tablet_manager.cc:1397] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250625 07:52:43.847805 1158 raft_consensus.cc:357] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.848057 1158 raft_consensus.cc:383] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.848193 1158 raft_consensus.cc:738] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2cdaf87b9d4b2a919098bf65efd513, State: Initialized, Role: FOLLOWER
I20250625 07:52:43.848445 1158 consensus_queue.cc:260] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.848320 596 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34788:
table { table_name: "default.Table_7a254ead6d2e4a3e9ede91b319970b00" } new_table_name: "default.Table_ae7a9fee81ee48b8a6b69dc8d1fc7791" modify_external_catalogs: true
I20250625 07:52:43.849150 1158 ts_tablet_manager.cc:1428] T ea62dfb8f20f4caeaa79ea1e85d99930 P 8f2cdaf87b9d4b2a919098bf65efd513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250625 07:52:43.849467 1158 tablet_bootstrap.cc:492] T 0bfa1c4d4b6e4530892118189a5ab516 P 8f2cdaf87b9d4b2a919098bf65efd513: Bootstrap starting.
I20250625 07:52:43.852674 1163 consensus_queue.cc:260] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } }
I20250625 07:52:43.854439 2877 raft_consensus.cc:491] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:52:43.854492 1046 raft_consensus.cc:3058] T 1796a3305cb94ccb9a0596643a736a96 P 8f2cdaf87b9d4b2a919098bf65efd513 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:52:43.854573 2877 raft_consensus.cc:513] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.855154 2877 leader_election.cc:290] T 47c9aca996884ffeab88381dd3cb189b P 191b611b772a4fda9d1cba98776138a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2cdaf87b9d4b2a919098bf65efd513 (127.0.104.3:36221), 429026e8d98d46339db143f84897a092 (127.0.104.2:45317)
I20250625 07:52:43.851833 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1492) took 2163 ms. Trace:
I20250625 07:52:43.856081 1160 rpcz_store.cc:276] 0625 07:52:41.688061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.688123 (+ 62us) service_pool.cc:225] Handling call
0625 07:52:43.838392 (+2150269us) ts_tablet_manager.cc:1145] Deleting tablet ac8968da295c4b7fb6d83b84935d27d2
0625 07:52:43.848733 (+ 10341us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.851811 (+ 3078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150017}
W20250625 07:52:43.857452 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet ac8968da295c4b7fb6d83b84935d27d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac8968da295c4b7fb6d83b84935d27d2 already in progress: creating tablet
I20250625 07:52:43.857973 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1493) took 2154 ms. Trace:
I20250625 07:52:43.858103 1160 rpcz_store.cc:276] 0625 07:52:41.703788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.703848 (+ 60us) service_pool.cc:225] Handling call
0625 07:52:43.857844 (+2153996us) ts_tablet_manager.cc:1145] Deleting tablet 363fee07295c44139fd713b33c82623c
0625 07:52:43.857854 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.857953 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2153807}
W20250625 07:52:43.858599 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 363fee07295c44139fd713b33c82623c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 363fee07295c44139fd713b33c82623c already in progress: creating tablet
I20250625 07:52:43.856469 1163 ts_tablet_manager.cc:1428] T 6e77ff1b8ca848ff8e1b4538645b4be2 P 429026e8d98d46339db143f84897a092: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250625 07:52:43.858943 1163 tablet_bootstrap.cc:492] T 550e64a19120444a8e7b27db9bbf09f5 P 429026e8d98d46339db143f84897a092: Bootstrap starting.
I20250625 07:52:43.859001 1160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34046 (request call id 1488) took 2183 ms. Trace:
I20250625 07:52:43.863111 1160 rpcz_store.cc:276] 0625 07:52:41.675485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0625 07:52:41.713839 (+ 38354us) service_pool.cc:225] Handling call
0625 07:52:43.858903 (+2145064us) ts_tablet_manager.cc:1145] Deleting tablet 487705c4b9ae4ab097771a69f03f05dc
0625 07:52:43.858911 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0625 07:52:43.858985 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144655}
I20250625 07:52:43.863790 1160 tablet_replica.cc:331] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: stopping tablet replica
W20250625 07:52:43.863750 583 catalog_manager.cc:4908] TS 429026e8d98d46339db143f84897a092 (127.0.104.2:45317): delete failed for tablet 487705c4b9ae4ab097771a69f03f05dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 487705c4b9ae4ab097771a69f03f05dc already in progress: creating tablet
I20250625 07:52:43.864097 1160 raft_consensus.cc:2241] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250625 07:52:43.864326 1160 raft_consensus.cc:2270] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250625 07:52:43.865056 1160 ts_tablet_manager.cc:1905] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250625 07:52:43.867473 1158 tablet_bootstrap.cc:654] T 0bfa1c4d4b6e4530892118189a5ab516 P 8f2cdaf87b9d4b2a919098bf65efd513: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.868415 1163 tablet_bootstrap.cc:654] T 550e64a19120444a8e7b27db9bbf09f5 P 429026e8d98d46339db143f84897a092: Neither blocks nor log segments found. Creating new log.
W20250625 07:52:43.869973 584 catalog_manager.cc:4908] TS 191b611b772a4fda9d1cba98776138a7 (127.0.104.1:46661): delete failed for tablet 7b3eafac1b2b46acb2ac6fca087b7f16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b3eafac1b2b46acb2ac6fca087b7f16 already in progress: creating tablet
I20250625 07:52:43.873068 1162 tablet_bootstrap.cc:654] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7: Neither blocks nor log segments found. Creating new log.
I20250625 07:52:43.884600 1162 tablet_bootstrap.cc:492] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7: No bootstrap required, opened a new log
I20250625 07:52:43.884829 1162 ts_tablet_manager.cc:1397] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.002s
I20250625 07:52:43.884996 593 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34788:
name: "default.Table_624198a39e244a768fef1d768a4c8a18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:52:43.885692 1162 raft_consensus.cc:357] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45317 } }
I20250625 07:52:43.885839 1160 ts_tablet_manager.cc:1918] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250625 07:52:43.885911 1162 raft_consensus.cc:383] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:52:43.885949 1160 log.cc:1199] T b782414a682246feb2811b4e3bf8b5a2 P 429026e8d98d46339db143f84897a092: Deleting WAL directory at /tmp/dist-test-taskSTcLfj/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750837927730000-416-0/minicluster-data/ts-1/wal/wals/b782414a682246feb2811b4e3bf8b5a2
I20250625 07:52:43.885977 1162 raft_consensus.cc:738] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191b611b772a4fda9d1cba98776138a7, State: Initialized, Role: FOLLOWER
W20250625 07:52:43.886106 593 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_624198a39e244a768fef1d768a4c8a18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:52:43.886232 1162 consensus_queue.cc:260] T 35a93e9198d1417bb1b8e00560921483 P 191b611b772a4fda9d1cba98776138a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2cdaf87b9d4b2a919098bf65efd513" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36221 } } peers { permanent_uuid: "191b611b772a4fda9d1cba98776138a7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46661 } } peers { permanent_uuid: "429026e8d98d46339db143f84897a092" member_type: VOTER last_known_addr { host: "127.0.104.2"